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BACKGROUND / PURPOSE 

The  Massachusetts  Comprehensive  Assessment  System  (MCAS),  as  a  key  component 
of  Massachusetts  Education  Reform,  was  conceptualized  and  designed  to  serve  as  a 
tool  to  support  efforts  of  schools  and  districts  to  improve  student  learning  by  providing  a 
consistent  measure  of  achievement  linked  to  the  statewide  curriculum  frameworks.  This 
system  assumes  use  of  the  results  by  educators  to  make  appropriate  changes  in 
curriculum,  instruction,  assessment,  and  allocation  of  resources.  As  in  other  states, 
different  aspects  of  this  strategy  are  under  discussion  on  many  levels  and  receiving 
praise  and  criticism  from  a  variety  of  stakeholders.  This  particular  research  study  has 
been  designed  and  conducted  with  a  relatively  narrow  focus  on  how  educators  in  the 
state  are  actually  making  use  of  the  MCAS  results.  It  does  not  address  whether  or  not 
the  changes  being  made  are  desirable  or  seek  to  understand  the  impact  that  reported 
changes  have  on  student  achievement. 

The  primary  purposes  of  this  research  study  were  to: 

•  Determine  to  what  extent  and  in  what  form  teachers  are  receiving  MCAS  results; 

•  Identify  common  practices  among  Massachusetts  schools  and  districts  in  utilizing 
MCAS  results  to  change  curriculum,  instruction,  assessment,  and  resource 
allocation; 

•  Identify  and  disseminate  information  about  positive  or  initially  successful  practices  in 
such  utilization,  as  well  as  any  "negative"  implications  of  changes  (should  they  exist), 
to  schools  throughout  the  Commonwealth;  and 

•  Promote  discussion  and  sharing  of  effective  strategies  for  using  MCAS  results 
among  participating  superintendents  and  other  district  administrators,  school 
principals,  and  teachers. 

RESEARCH  QUESTIONS 

The  following  is  a  sampling  of  research  questions  addressed  in  this  study  from  the 
variety  of  data  collection  methods  used. 

•  What  information  about  MCAS  performance  are  schools  and  districts  disseminating 
to  their  classroom  teachers?  What  methods  are  they  using  to  disseminate  the 
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information  and  at  what  level  of  aggregation  and  detail?  To  what  extent  and  in  what 
ways  are  teachers  and  principals  engaged  in  analysis  and  discussion  of  MCAS 
student  performance  data  related  to  district,  school,  and  individual  teacher 
accountability? 

Are  schools  and  districts  using  their  MCAS  results  to  analyze  performance  and 
implement  changes  in  curriculum,  instruction,  assessment,  and/or  resource 
allocation?  If  so,  how  are  they  going  about  these  changes  and  who  is  involved?  To 
what  extent,  if  any,  are  current  courses  and  programs  being  eliminated  or  modified 
in  negative  ways  as  a  result  of  MCAS  related  decisions?  How  are  teachers  being 
drawn  into  this  process?  Which  strategies  are  gaining  teacher  buy-in  and  perceived 
empowerment  through  the  constructive  use  of  the  MCAS  as  opposed  to  forced  use 
for  change? 

What  forms  of  assistance  and/or  professional  development  are  being  made  available 
and  are  useful  in  helping  teachers  understand  the  information  provided  and  how  to 
use  it?  What  incentives  are  in  place  for  teachers  to  use  this  information  to  change 
instructional  practice  in  their  classrooms?  To  what  extent  and  in  what  ways  do 
administrators  and  teachers  engage  in  productive  discussions  of  MCAS  results 
focused  on  student  identified  needs?  What  professional  development  practices  and 
models  have  helped  to  foster  productive  MCAS  data  analysis,  discussion  and 
modification  of  instructional  practice?  To  what  extent  is  analysis  of  MCAS  data 
serving  as  a  stimulus  for  groups  of  educators  in  individual  schools  to  work  together 
toward  common  purposes  based  on  identified  student  needs? 

How  are  teachers  actually  using  this  information  to  change  instructional  and 
assessment  practice  in  their  classrooms?  What,  if  any,  results  have  been  observed 
as  a  result  of  these  changes  in  practice?  Are  teachers  able  to  articulate  the 
connections  between  the  changes  they  are  making  and  the  MCAS  results?  Do 
teachers  believe  the  changes  they  are  making  in  practice  are  designed  to  enhance 
student  learning,  as  well  as  student  test  taking  ability? 

How  are  superintendents  and  other  district  administrators  exerting  leadership  in  the 
use  of  MCAS  for  school  improvement  through  their  work  with  school  principals, 
School  Committees,  teachers,  parents  and  other  community  members?  What 
communication  and  professional  development  strategies  are  they  using  and  finding 
to  be  most  effective  in  encouraging  learning  and  positive,  responsible  use  of  MCAS 
results  within  individual  schools  and  the  district  as  a  whole?  Which  districts  are 
making  the  most  effective  use  of  their  MCAS  data  and  how  are  they  using  this 
information? 

What  strategies  are  school  principals  using  to  engage,  motivate,  and  support 
teachers  to  use  the  MCAS  to  make  changes  to  curriculum,  instruction,  and 
assessment?  What  changes  in  budget  allocations  are  being  made  to  support  this 
work?  How  are  these  budget  allocation  decisions  being  made?  How  are  school 
principals  preparing  themselves  to  better  understand  their  schools'  MCAS 
information  and  to  articulate  resulting  plans  to  address  identified  needs? 
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METHODS 

The  following  data  collection  activities  were  used  in  preparation  for  this  report: 

Stage  1  Data  Collection  -  During  this  initial  stage  (late  January  and  early  February, 
2000),  Learning  Innovations  conducted  a  series  of  focus  group  discussions  for 
superintendents,  principals,  and  teachers  in  three  locations  (Stoneham — north  of 
Boston,  Plymouth — south  of  Boston,  and  Springfield — in  the  western  part  of  the  state). 
The  educators  invited  to  the  focus  groups  were  selected  through  a  stratified  random 
sampling  procedure,  ensuring  equal  numbers  of  districts  invited  from  each  of  the  six 
geographical  regions  identified  on  the  Massachusetts  Department  of  Education  (DOE) 
database.  Invitation  letters  for  focus  groups  were  sent  to  superintendents  from  25 
districts  for  each  of  the  three  locations.  Letters  were  also  sent  to  the  principals  of 
randomly  selected  elementary,  middle,  and  high  schools  (one  of  each  level  from  the 
same  districts)  inviting  them  to  attend  a  focus  group  at  the  same  location  at  a  different 
time.  Principals  were  asked  to  share  and  post  focus  group  invitations  for  teachers  in 
their  schools.  Invitation  letters  explained  the  purpose  and  plan  for  the  research  study. 
Follow-up  calls  were  made  to  encourage  attendance.  Each  focus  group  (8-15 
participants)  included  a  90-minute  discussion  with  a  facilitator  and  notetaker  present. 
Focus  group  discussions  were  also  tape  recorded. 

This  strategy  of  beginning  with  in-depth  focus  group  discussions  was  used  in  order  to: 
provide  opportunities  for  educators  to  participate  in  thought-provoking  discussions  in 
which  they  were  able  to  share  and  reflect  on  their  experiences  with  change-making 
using  MCAS;  and  to  inform  the  development  of  questionnaires  that  were  distributed 
during  Stage  2  to  a  broader,  statewide  sample  of  these  groups.  The  focus  group 
participants  also  served  as  rich  resources  for  beginning  to  identify  and  gather  more  in- 
depth  information  concerning  practices  in  schools  and  districts  that  the  educators  polled 
deemed  to  be  effective. 

Stage  2  Data  Collection  -  Based  on  collaborative  discussions  with  the  funding 
organization  and  findings  of  the  extensive  focus  group  work,  questionnaires  were 
developed  and  piloted.  Finally,  the  questionnaires  were  distributed  in  the  following 
manner. 

•  All  superintendents  in  the  state  received  a  questionnaire  to  complete. 

•  A  stratified  random  sample  of  districts  was  then  drawn,  identifying  120  school 
districts  (and  combined  school  districts)  from  six  different  geographical  areas 
(determined  by  the  Massachusetts  Department  of  Education  listing).  From  the  DOE 
listing  of  schools  in  the  state,  another  sub-sample  was  drawn  randomly  selecting 
one  school  from  each  level  (elementary,  middle,  and  high)  contained  within  that 
district. 

•  Questionnaires  were  then  sent  in  care  of  the  superintendent  in  the  selected  districts, 
requesting  that  questionnaires  or  questionnaire  packets  be  delivered  to: 
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•  Three  district  curriculum  directors/supervisors  where  that  position  exists;  and 

•  The  three  selected  schools  (questionnaires  for  the  principal  and  three  teachers  in 
each  school). 

Stage  3  of  Data  Collection  -  The  researchers  conducted  telephone  interviews  and  site 
visits  to  gather  additional  information  about  effective  strategies  for  using  MCAS  for 
school  and  district  improvement  from  educators  who  volunteered  through  the  focus 
groups  or  questionnaires.    Well  over  100  calls  were  made,  as  well  as  numerous  site 
visits. 

Stage  4:  Report  and  Dissemination  of  Information  about  Initially  Successful 
Practices  -  In  addition  to  this  written  research  report,  a  fall  dissemination  session  is 
planned  for  sharing  of  the  findings  of  this  study  with  educators  throughout  the  state. 


FINDINGS 

The  Findings  section  of  this  report  presents  the  findings  from  the  focus  groups, 
telephone  interviews,  site  visits,  and  the  four  sets  of  questionnaires  (for 
superintendents,  curriculum  directors,  principals,  and  teachers).  See  Appendix  A  for 
blank  questionnaires  and  a  complete  list  of  districts  from  which  questionnaires  were 
received. 

Educators  from  36  districts  participated  in  focus  group  discussions  held  in  Stoneham, 
Plymouth,  and  Springfield.  Focus  group  participants  included  21  superintendents,  21 
principals,  and  17  teachers.  A  list  of  participating  districts  appears  in  Appendix  B. 

Completed  and  usable  questionnaires  were  received  from: 

•  1 31  of  the  superintendents,  for  a  44%  response  rate. 

•  1 66  principals  from  87+  districts.  (Some  principals  did  not  identify  their  districts.) 
This  comes  to  73%  of  the  districts  in  the  sample  that  submitted  principal 
questionnaires.  The  percentage  response  of  principals  was  46%  (166  of  360). 

•  86  curriculum  directors  from  50+  districts.  (Some  curriculum  directors  did  not  identify 
their  districts.)  This  comes  to  42%  of  the  districts  in  the  sample  that  submitted 
curriculum  director  questionnaires.  The  percentage  response  of  curriculum  directors 
was  24%  (86  of  360).    Please  note  that  many  districts  do  not  have  curriculum 
director  positions. 

•  379  teachers  from  91  districts.  (Some  teachers  did  not  identify  their  districts.) 
This  comes  to  76%  of  the  districts  in  the  sample  that  submitted  teacher 
questionnaires.  The  percentage  response  of  teachers  was  37%  (397  of  1080). 

It  should  be  noted  that  the  participation  of  teachers  both  in  focus  groups  and 
questionnaire  completion  was  lower  than  hoped,  with  no  clear  explanation  except  the 
many  priorities  competing  for  their  time. 
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Overall,  completed  questionnaires  were  received  from  a  total  of  1 55  identified  school 
districts  (and  a  few  that  were  unidentified),  yielding  responses  from  about  half  the 
districts  in  the  state.  Sample  representation  from  across  the  six  Department  of 
Education  (DOE)  geographical  regions,  as  well  as  across  the  DOE's  seven  categories 
of  school  communities  was  very  close  to  the  population  breakdowns  of  districts  across 
the  state.  See  Tables  1  and  2  below. 


Table  1:  Distribution  of  Sample  by  Geographical  Regions 


GEOGRAPHICAL 
REGION 

PERCENT  OF 

DISTRICTS 

RESPONDING  IN 

SAMPLE 

PERCENT  OF 

OPERATING 

DISTRICTS  IN  THE 

STATE 

Greater  Boston 

17% 

14% 

Northeast 

21% 

16% 

Central 

22% 

24% 

Southeast 

23% 

25% 

Springfield  Area 

10% 

10% 

Pittsf  ield  Area 

7% 

11% 

Table  2:  Distribution  of  Sample  by  DOE  Community  Description  Categories 


TYPE  OF  COMMUNITY 


PERCENT  OF 

DISTRICTS 

RESPONDING  IN 

SAMPLE 


PERCENT  OF 

OPERATING 

DISTRICTS  IN  THE 

STATE 


Urbanized  Center 


22% 


15% 


Economically  Developed 
Suburbs 


24% 


21% 


Growth  Communities 


14% 


16% 


Residential  Suburbs 


19% 


17% 


Rural  Economic  Centers 


14% 


16% 


Small  Rural 
Communities 


5% 


7% 


Resort/Retirement  and 
Artistic 


3% 


9% 


In  addition,  the  list  of  districts  from  which  questionnaires  were  received  was  compared 
to  analyses  of  1 999  MCAS  results  reported  in  the  Boston  Globe  in  December  1 999. 
Thirty-four  of  the  responding  districts  were  listed  among  the  50  districts  from  across  the 
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state  that  showed  the  greatest  improvement  in  MCAS  scores  from  1998.  These  34 
districts  made  up  22%  of  the  overall  sample  of  districts  that  responded  to  the  survey. 

Another  analysis  was  presented  in  the  Globe,  ranking  the  210  districts  in  the  state  that 
administered  MCAS  in  English,  mathematics,  science  and  technology  in  all  three 
grades.  One  hundred  and  twenty-nine  of  the  districts  responding  to  the  survey  were 
included  in  this  analysis,  with  almost  identical  percentage  of  responding  districts  in  the 
top  (35%),  middle  (30%),  and  bottom  (37%)  rankings,  indicating  that  responding  districts 
were  almost  evenly  distributed  across  these  overall  categories  of  student  achievement 
scores. 

The  findings  from  all  sources  of  data  are  organized  in  the  following  sections: 
Changes  Being  Made  Based  on  MCAS  Results 
Data  Preparation,  Dissemination 
District-Level  Communication,  Incentives,  and  Leadership 
Data  Use  Processes 
Challenges  Facing  Schools  and  Districts. 


FINDINGS:  Changes  Being  Made  Based  on  MCAS 
Results 

Finding  #1:  Over  90%  of  alt  groups  sun/eyed  indicated  that  curricular  changes 
have  been  made  or  identified  based  on  MCAS  results,  with  lesser  percentages 
noting  changes  in  instruction,  assessment,  and  allocation  of  resources. 

When  asked  if  changes  in  curriculum  had  been  made  or  identified  based  on  MCAS 
results,  over  90%  of  all  four  groups  indicated  that  such  changes  had  been  made. 
Proportions  of  these  groups  indicating  changes  in  instruction  ranged  from  87%  for 
curriculum  directors  to  92%  of  principals.  Results  concerning  assessment  changes 
hovered  around  80%,  whereas  the  percentages  of  these  groups  indicating  changes  in 
allocation  of  resources  were  much  lower,  with  the  exception  of  superintendents  of  whom 
83%  noted  changes  made  in  this  area.  See  Table  3. 
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Table  3:  Types  of  Changes  Made  Based  on  MCAS  Results* 


TYPE  OF  CHANGE 

MADE  OR  IDENTIFIED 

BASED  ON  MCAS 

RESULTS 

%  OF 
SUPTS. 

%OF 

CURRICULUM 

DIRECTORS 

%OF 
PRINCIPALS 

%OF 
TEACHERS 

Changes  in 
Curriculum 

98% 
n=130 

93% 
n=80 

95% 
n=162 

93% 
n=367 

Changes  in 
Instruction 

91% 
n=123 

87% 
n=82 

92% 
n=151 

88% 
n=343 

Changes  in 
Assessment 

82% 
n=119 

80% 
n=75 

83% 
n=153 

78% 
n=344 

Changes  in  Allocation 
of  Resources 

83% 
n=119 

62% 
n=79 

67% 
n=159 

55% 
n=317 

*  Some  respondents  did  not  answer  all  questions.  Therefore  the  n's  are  slightly  variable. 

Finding  #2;  Low  scores  in  general  or  in  specific  curricular  areas  as  well  as  low 
scores  on  open  response  items  and  writing  compositions  were  cited  as  findings 
prompting  changes  made  in  curriculum,  instruction,  assessment,  and  allocation 
of  resources. 

All  survey  respondents  were  asked  to  indicate  specific  MCAS  results  that  prompted  the 
changes  they  described  in  the  areas  of  curriculum,  instruction,  assessment,  and 
allocation  of  resources.  The  most  frequent  response  for  all  four  groups  was  simply  low 
scores  in  general  (or  lower  than  desired).  Lesser  proportions  of  the  various  groups 
mentioned  lower  than  desired  scores  in  specific  areas  such  as  math,  English/language 
arts,  and  science. 

Also  cited  by  many  of  the  respondents  were  the  MCAS  results  for  open  response  items. 
This  was  a  particularly  important  finding  that  prompted  instructional  changes  according 
to  the  educators  surveyed  (ranging  from  21%  to  36%  in  the  four  groups),  and  likewise,  a 
result  that  prompted  changes  in  classroom  assessment  practices  according  to  a  range 
of  28%  to  35%  of  the  different  groups  surveyed.  Many  educators  in  the  surveys  and  in 
the  focus  groups  noted  insights  they  gained  from  conducting  item  analysis  of  MCAS 
scores  as  well  as  studying  the  general  content  and  format  of  the  tests  themselves. 


Finding  #3;  Decision-making  about  curricular  and  other  changes  commonly 
involves  collaborative  team  efforts  among  teachers,  principals,  curriculum 
directors,  and  central  administrators. 

For  all  four  areas  of  change,  the  greatest  proportion  of  each  of  the  four  responding 
groups  of  educators  indicated  the  involvement  of  school  administrators  and  teachers, 
and  to  a  somewhat  lesser  degree  the  work  of  curriculum  coordinators  and  central 
administrators  including  superintendents  and  assistant  superintendents.  Guidance 
counselors  were  rarely  mentioned  as  being  involved  in  decisions  being  made.  A  variety 
of  team  or  committee  configurations  were  mentioned  including  grade  level  teams, 
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A  Positive  Practice  from  the  Field:  A  high  school  team  attended  a  prof  essional 
development  session  with  an  outside  organization  focusing  on  data  analysis.  Team 
members  indicated  that  they  benefited  from  exposure  to  ways  other  schools  and 
districts  were  doing  MCAS  item  analysis.  Subsequent  department  meetings  at  the 
high  school  level  focused  on  curriculum  mapping,  aligning  curriculum  to  standards, 
analyzing  tests  and  answers,  and  linking  MCAS  questions  with  learning  standards. 
Close  analysis  of  questions  and  student  responses  yielded  summaries  of  content  gaps 
and  curriculum  modifications  needed.  All  9th  and  10  grade  teachers  created  tACAS 
preparation  and  remediation  plans  linked  to  instruction  (increased  writing,  rubric  use, 
focus  on  open  response  questions).  Further  teamwork  in  conjunction  with  the 
guidance  department  resulted  In  development  of  student  prof  iles  for  ninth  graders  in 
failing  and  needs  improvement  categories,  which  include  specif  ic  instructional 
recommendations.  Student  analysis  forms  were  used  to  document  current 
information  and  recommendations  for  teacher  instruction  and  follow  up  courses  to 
improve  student  results.  Remedial  courses  were  then  implemented  for  8th  and  10th 
grade  students.  Plans  for  further  improvements  include  professional  development  in 
writing  standards-based  curriculum/instruction  unit  plans,  using  student  profiles  and 
test  question  analysis,  and  writing  monthly  instructional  follow-up  reports  and  unit 
plans. 


Finding  #4;  Most  commonly  cited  changes  made  in  curriculum,  instruction,  and 
assessment  based  on  MCAS  results  were  indicated  in  English/Language  Arts, 
mathematics,  and  writing.  Most  commonly  cited  changes  in  allocation  of 
resources  included  providing  professional  development  and  curriculum  work, 
obtaining  new  textbooks  and  materials,  providing  MCAS  support  for  students, 
and,  to  some  extent,  hiring  of  staff. 

Curriculum  changes  —  The  most  common  types  of  significant  curriculum  changes 
cited  as  examples  in  open-ended  questions  were  indicated  in  the  areas  of 
English/language  arts,  mathematics,  and  writing  across  the  curriculum  and  use  of 
graphic  organizers.  Others  mentioned  included  test  taking  skills  and  MCAS  support  for 
students.  See  Table  4  below. 
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Table  4:  Most  Commonly  Cited  Curriculum  Changes' 


EXAMPLES  OF 

%OF 

%OF 

%OF 

%OF 

SIGNIFICANT 

SUPTS 

CURRICULUM 

PRINCIPALS 

TEACHERS 

CURRICULUM 

ri=124 

DIRECTORS 

n=148 

n=342 

CHANGE:; 

n=76 

•     ELA  change 

20% 

16% 

7% 

10% 

•     Math  change 

18% 

14% 

23% 

13% 

•     Science  change 

8% 

8% 

9% 

9% 

•     Social  studies 

change 

8% 

4% 

8% 

7% 

•     Writing  across  the 

curriculum  and  use 

of  graphic 

organizers 

12% 

14% 

21% 

19% 

•     Use  of  more  open 

response 

questions 

6% 

15% 

•     Test  taking  skills, 

including 

vocabulary 

13% 

3% 

14% 

•      MCAS  support 

classes  or  tutoring 

during  school, 

after  school, 

summer,  weekends 

2% 

8% 

19% 

*Not  all  respondents  who  indicated  changes  made,  e.g.,  in  curriculum,  provided  examples  of  these 
changes. 

Questionnaire  and  focus  group  results  showed  considerable  changing  of  sequence  of 
courses,  especially  in  mathematics,  English,  and  social  studies,  as  well  as  deletion  of 
courses  in  industrial  arts,  home  economics,  and  general  math  and  business  courses. 
Grade  9  courses  in  technology  were  cited  frequently  as  an  addition.    See  Table  5 
below. 
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Table  5:  Most  Commonly  Cited  Curricular  Areas  in  Which  Courses  Were  Added 
or  Eliminated 


COURSES  ADDED 

%  OF 

SUPTS. 

n=74 

%OF 

CURRICULUM 

DIRECTORS 

n=39 

%OF 

PRINCIPALS 

n=96 

■  :         -.■.'■'■ 

%OF 

TEACHERS 

n=367 

ELA 

15% 

15% 

19% 

13% 

Math 

24% 

28% 

25% 

21% 

Science 

7% 

5% 

13% 

12% 

Social  Studies 

18% 

13% 

21% 

15% 

Technology 

8% 

10% 

7% 

3% 

MCAS  math 

8% 

5% 

2% 

MCAS  science 

3% 

3% 

MCAS  ELA 

3% 

8% 

2% 

MCAS  support — 
other 

3% 

23% 

Foreign  Language 

4% 

5% 

2% 

COURSES 
ELIMINATED 

%  OF 
SUPTS. 

n=49 

%  OF 

CURRICULUM 

DIRECTORS 

n=29 

%  OF 

PRINCIPALS 

n=65 

%  OF 

TEACHERS 

n=94 

ELA 

10% 

2% 

3% 

Math 

22% 

24% 

23% 

22% 

Science 

8% 

10% 

12% 

11% 

Social  Studies 

18% 

17% 

14% 

14% 

MCAS  ELA 

15 

Industrial  arts,  home 
economics 

4% 

3% 

11% 

5% 

Electives 

4% 

7% 

3% 

4% 

Other,  e.g.,  business, 
general  courses 

18% 

14% 

6% 

8% 

Material  was  added  to  or  eliminated  from  existing  courses  or  curricula  in  order  to  match 
MCAS  test  requirements,  predominantly  in  the  areas  of  mathematics,  English/language 
arts,  science,  and  social  studies.  See  Table  6  below. 
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Table  6:  Most  Commonly  Cited  Curricular  Areas  within  which  Material  Was  Added  or  Eliminated 


MATERIAL  ADDED 

WITHIN  THESE 

CURRICULAR  AREAS 

%  OF 
SUPTS. 

N=40 

%OF 

CURRICULUM 

DIRECTORS 

n=27 

%OF 

PRINCIPALS 

n=56 

%OF 
TEACHERS 

n=109 

ELA 

15% 

7% 

9% 

8% 

Math 

10% 

7% 

14% 

16% 

Science 

3% 

4% 

14% 

16% 

Social  Studies 

23% 

26% 

16% 

17% 

Technology 

10% 

15% 

7% 

5% 

MCAS  general  support 

26% 

42% 

4% 

Study  skills 

13% 

Writing  across 
curriculum 

7% 

14% 

8% 

Foreign  Language 

5% 

4% 

2% 

Other,  e.g.,  problem- 
solving,  new  books 

13% 

4% 

4% 

15% 

None  yet 

18% 

11% 

23% 

8% 

MATERIAL 

ELIMINATED  WITHIN 

THESE  CURRICULAR 

AREAS 

%  OF 
SUPTS. 

N=32 

%OF 

CURRICULUM 

DIRECTORS 

n=17 

%OF 

PRINCIPALS 

h=41 

%OF 
TEACHERS 

n=53 

ELA 

6% 

Math 

3% 

12% 

5% 

6% 

Science 

3% 

18% 

7% 

11% 

Social  Studies 

19% 

18% 

7% 

21% 

None  yet 

41% 

18% 

54% 

32% 

A  Positive  Practice  from  the  Field:  Teachers  in  a  middle  school  met  by  subject  area. 
They  analyzed  student  responses  on  MCAS  open-ended  questronsqnd  then  made 
recommendations  for  curriculum  modifications,  specif ically  in  the  areas  of  drama  and 
earth  science.  Teachers  Indicated  that  through  their MCAS  analysis  and  curriculum 
development  that  they  better  understood  each  other's  areas  of  expertise  and  that 
collaboration  within  teaching  teams  improved.  y     .  . . :, 
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A  Positive  Practice  from  the  Field:  Item  analysis  of  questions  by  a  team  of  teachers  and 
an  administrator  at  the  high  school  level  identified  areas  of  weakness  and  necessary 
curriculum  adjustments.  As  a  result  of  this  work,  staff  took  action  to: 

•  address  vocabulary  weaknesses  in  test  taking  skills; 

•  respond  to  the  large  number  of  technology  questions  on  test  by  adding  a  mandatory 
technology  class  for  all  freshmen; 

•  eliminate  Civics  and  replace  it  with  American  Government  in  senior  year; 

•  implement  freshman  and  sophomore  levels  of  World.  History  I  and  H; 

•  add  an  integrated  science  class; 

•  eliminate  consumer  math  and  transitional  math  classes; 

•  assure  that  al I  students  take  Algebra  I  and  Geometry;  and 

•  adjust  student  schedules  to  include  additional  daily  classes  to  meet  the  needs  of 
different  learners. 

Instructional  Changes  -  When  educators  were  asked  in  the  survey  about  changes 
made  in  instruction,  once  again,  the  greatest  numbers  indicated  changes  in  writing 
instruction  including  use  of  graphic  organizers,  writing  across  the  curriculum  and  use  of 
more  open  response  questions.  Other  changes  mentioned  less  commonly  included 
increase  in  math  problem  solving,  using  a  greater  variety  of  teaching  strategies,  and 
general  MCAS-related  activities.  See  Table  7  below. 


7:  Most  Commonly  Cited 

Changes 

in  Instruction* 

EXAMPLES  OF 

%OF 

%  OF 

%  OF 

%OF 

SIGNIFICANT 

SUPTS. 

CURRICULUM 

PRINCIPALS 

TEACHERS 

INSTRUCTIONAL 

DIRECTORS 

CHANGES: 

n=  97 

n=67 

n=131 

n=310 

Math  inquiry, 
problem  solving, 
projects 

7% 

4% 

3% 

1% 

Writing,  writing 
across  the 

curriculum,  using 

graphic  organizers 
for  writing 

21% 

39% 

36% 

27% 

Use  of  more  open 

response  questions 

14% 

12% 

25% 

20% 

Use  of  rubrics  with 

students 

3% 

6% 

10% 

4% 

Greater  variety  of 
teaching  strategies 

9% 

1% 

2% 

MCAS-related 

activities,  e.g., 

homework, 

questions,  general 
support 

1% 

6% 

4% 

12% 

All  students  have 

same  texts  and 

instruction 

7% 

1% 

*Not  all  respondents  who  indicated  changes  made,  e.g.,  in  instruction,  provided  examples  of  these  changes. 
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A  Positive  Practice  from  the  Field:  A  ninth  grade  English  teacher  reports  positive 
results  in  the  classroom  with  an  instrument  created  to  help  students  approach  open- 
response  questions.  This  teacher  modified  an  existing  evaluation  instrument' that  uses 
mnemonics  to  enable  students  to  respond  to  open-ended  questions  with  a  step-by-step 
approach  that  allows  them  to  gain  the  best  possible  score  (and  know  they  have  achieved  it) 
by  addressing  all  components  of  an  optimum  response.  This  teacher  also  created  MCAS 
Study  Guides  with  test  questions  from  prior  MCAS  tests  and  shared  them  with  other 
teachers. 


A  Positive  Practice  from  the  Field'-  A  middle  school  teacher  of  an  alternative  class  of  at- 
risk  students  uses  writing  prompts  similar  to  those  on  the  MCAS  long  compositions. 
Students  write  daily  essays  and  then  self  assess  and  assess  peer:essays  using  A\CAS 
writing  rubrics.  Students  reportedly  gain  in  writing  skill;  understanding  of  bhCAS 
expectations,  and  confidence  in  producing  high  quality  compositions. 

Assessment  Changes  -  The  largest  proportion  of  all  respondent  groups  (ranging  from 
27%  to  35%)  indicated  the  use  of  rubrics  with  students  (especially  focused  on  writing) 
as  an  example  of  changes  made  in  classroom  assessment  based  on  MCAS  results. 
Roughly  a  quarter  of  each  group  noted  using  more  open  response  items,  and  many 
mentioned  using  more  performance-based  assessments,  benchmarks,  or  portfolios,  as 
well  as  emphasizing  more  frequent  assessment  of  student  performance.  Some  mention 
was  made  as  well  of  recently  developed  reading  assessments,  more  writing  and 
problem  solving  in  math,  and  generally  developing  assessments  that  mirror  the  MCAS. 
See  Table  9below. 


Table  8:  Most  Commonly  Cited  Changes  in  Assessment* 


EXAMPLESOF 
SIGNIFICANT 
CHANGES  IN 

ASSESSMENT 

%  OF 

SUPTS. 

n=104 

%OF 

CURRICULUM 

DIRECTORS 

h=55 

%OF 
PRINCIPALS 
1n=126 

%OF 

TEACHERS 

n=257 

Performance-based 
tasks  /  benchmarks  / 
portfolios 

21% 

16% 

6% 

12% 

Rubrics 

28% 

27% 

35% 

37% 

More  Open  Response 

22% 

24% 

21% 

28% 

New  or  Developed 
reading  assessments 

4% 

2% 

8% 

<1% 

Fewer  multiple  choice, 
one-word,  recall  type 
items 

4% 

8% 

1% 

Assessments  that 
mirror  MCAS 

9% 

5% 

More  writing  and 
problem  solving  in 
math 

1% 

2% 

2% 

4% 

*Not  all  respondents  who  indicated  changes  made,  e.g.,  in  assessment,  provided  examples  of  these 
changes. 
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A  Positive  Practice  from  the  Field:  A  district  team  created  a  standards-based  rubric  for 
the  writing  improvement  process.  They  then  gave  every  student's  long  composition  to 
former  and  present  teachers  who  then  analyzed  the  compositions  with  the  rubric.  The 
teachers  indicated  that  this  work  with  actual  writing  samples  improved  their  ability  to 
identify  their  students'  writing  strengths  and  weaknesses.        . 


i — 


A  Positive  Practice  from  the  Field:  A  district-wide  effort  was  launched  to  ensure  that 
all  students  (not  characterized  as  students  with  specials  needs)  have  a  personalized 
education  plan.  This  work  focused  first  on  students  (K-12)  in  the  lowest  quartile  and  those 
who  had  not  scored  wellon  the  MCAS  tests.  Principals  and  Staff  completed  an  in-depth 
analysis  of  each  student;  documenting  the  student's:  strengths  and  areas  of  need  (linked 
to  learning  standards);  current  level  of  performance;  approach  tp  learning;  and  optimum 
learning  setting.  They  then  identified  strategies  to  help  the  student  improve;  planned 
follow  up  assessment;  notified  parents;  and  informally  piloted  the  systems  Teachers 
indicate  that  they  find  the  plan  useful,  providing  a  solid  frame  of  reference  for  their 
continuing  work  with  students. 

Allocation  of  Resources  -  The  predominant  areas  of  changes  in  allocation  of 
resources  based  on  MCAS  results  are  in  professional  development  and  curriculum 
work,  obtaining  new  textbooks  and  materials,  providing  MCAS  support  for  students 
(during,  before,  and  after  school),  and  to  some  degree,  hiring  of  staff.    Some  focus 
group  participants  indicated  increased  resources  allocated  to  perform  additional  data 
analysis,  e.g.,  correlation  of  MCAS  scores  with  other  available  data  such  as  attendance, 
age,  and  grades  of  students.  See  Table  9  below. 

Table  9:  Most  Commonly  Cited  Examples  of  Significant  Changes  in  Allocation  of 
Resources* 


%  OF 

SUPTS. 

n=94 

%  OF 

CURRICULUM 

DIRECTORS 

n=48 

%OF 

PRINCIPALS 

n=68 

%OF 

TEACHERS 

n=166 

New  textbooks  / 
materials 

29% 

44% 

34% 

47% 

Professional 
development  and 
curriculum  work 

30% 

33% 

16% 

19% 

MCAS  support  for 
students 

27% 

7% 

2% 

Hiring  of  staff 

7% 

10% 

3% 

9% 

•Not  all  respondents  who  indicated  changes  made,  e.g.,  in  allocation  of  resources,  provided 
examples  of  these  changes. 
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Finding  #5;  Many  survey  respondents  say  that  it  is  too  soon  to  determine  the 
results  of  curricular  and  other  changes,  however,  some  indicate  MCAS  score 
improvement,  increases  in  student  writing  activities  and  skill  levels,  and 
increases  in  student  and  teacher  awareness  and  acceptance  of  the  value  of 
MCAS. 

Roughly  one  third  of  all  survey  groups  indicated  that  it  was  too  soon  to  tell  what  the 
results  are  of  the  changes  implemented  in  curriculum,  instruction,  assessment,  and 
allocation  of  resources.  However,  some  (2%-20%  of  the  various  groups)  did  indicate 
increase  in  MCAS  scores  that  they  attributed  to  changes  made.  Other  results  mentioned 
most  often  were  more  writing,  improved  writing,  and  improved  open-ended  responses. 

Other  results  from  these  changes  gleaned  from  surveys  and  focus  groups  include 
increased  awareness  of  the  importance  of  MCAS;  increased  student  and  teacher 
understanding  of  MCAS  expectations;  improved  student  attitudes  concerning  MCAS, 
e.g.,  willingness  to  try,  confidence  in  test  taking;  increased  communication  and 
collaboration  among  teachers  stemming  from  MCAS  related  work;  improved  instruction 
focused  on  the  needs  of  students;  and  use  of  a  greater  variety  of  assessment  strategies 
in  classrooms. 


FINDINGS:  Data  Preparation,  Dissemination 

Finding  #6:  Data  analysis  responsibilities  are  shared  across  roles  in  districts, 
particularly  among  central  office  personnel,  principals,  curriculum  directors,  and 
school  and  subject  area  teams  of  teachers. 

Superintendents  and  curriculum  directors  in  describing  data  analysis  responsibilities 
indicated  that  this  was  a  responsibility  shared  by  many.  Seventy-six  percent  of  the 
superintendents  surveyed  said  that  they  were  responsible  for  data  analysis,  as  well  as 
82%  indicating  responsibility  of  the  principals  and  over  60%  mentioning  this  as  a 
responsibility  of  school  and  subject  area  teams.  Seventy-one  percent  of  curriculum 
directors  indicated  their  own  responsibility  for  data  analysis,  as  well  as  71  %  indicating 
principals  and  59%  mentioning  subject  area  teams. 

Principals  (97%)  and  teachers  (79%)  pointed  to  the  responsibility  of  the  principal  in  this 
domain.  About  half  of  each  group  indicated  the  responsibility  of  subject  area  teams  of 
teachers  and  39%  of  teachers  and  49%  of  principals  indicating  responsibility  of  grade 
level  teams. 

When  asked  if  they  had  been  personally  involved  in  analyzing  the  MCAS  results,  99% 
of  curriculum  directors,  95%  of  principals,  82%  of  teachers,  and  76%  of  the 
superintendents  indicated  that  they  had.  See  Table  10. 
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Table  10:  Percent  of  Educators  in  Four  Groups  That  Personally  Analyzed  MCAS 
Results* 


CONSTITUENCY 

%  YES 

%NO 

Superintendents,  n  =  131 

76% 

24% 

Curriculum  Directors,  n  =  85 

99% 

1% 

Principals,  n  =  165 

95% 

5% 

Teachers,  n  =  339 

82% 

18% 

Some  respondents  did  not  answer  all  questions.  There 


ore  the  n's  are  slightly  variable. 


Finding  #  7:  MCAS  results  are  being  disseminated  to  the  school  level  to 
principals  and  teachers  with  data  disaggregated  most  commonly  by  grade  level, 
curriculum  area,  sub-scores  by  curriculum  areas,  and  test  item,  and  less 
frequently  by  race/ethnicity,  gender,  or  feeder  school. 

One  hundred  percent  of  the  responding  superintendents  indicated  that  they  had 
provided  the  data  to  all  the  schools  in  their  districts,  with  98%  to  the  principals  and  68% 
to  all  faculty/staff.  Similar  responses  were  received  from  the  curriculum  directors, 
principals,  and  teachers,  indicating  receipt  of  the  data  from  their  superintendents. 

The  four  groups  of  respondents  were  consistent  in  their  responses  about  the  level  of 
analysis  of  their  districts'  results,  with  over  90%  of  all  indicating  analysis  by  grade  level 
and  curriculum  area.  Half  of  the  teachers  and  69%  to  76%  of  principals,  curriculum 
directors,  and  superintendents  indicated  the  provision  of  sub-scores  in  the  curriculum 
areas.  Less  than  15%  of  all  groups  said  they  received  breakdowns  in  the  areas  of 
race/ethnicity  or  gender.  (Note  that  the  lack  of  breakdown  by  race/ethnicity  may  be 
related  to  low  diversity  in  some  districts.)  Roughly  two-thirds  to  three-quarters  of  each 
of  the  groups  mentioned  results  analyzed  by  school. 

Over  90%  of  each  group  indicated  school  results  broken  down  by  grade  level  and 
curriculum  areas,  70-80%  of  all  groups  pointing  to  breakdowns  by  curriculum  sub- 
scores  and  test  items.  The  percentages  were  again  low  by  gender  and  race/ethnicity, 
as  well  as  in  the  area  of  analysis  by  feeder  schools.  See  Table  1 1 . 
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LEVEL  OF  ANALYSIS 

%  OF 
SUPTS. 

%OF 

CURRICULUM 

DIRECTORS 

%OF 
PRINCIPALS 

%OF 
TEACHERS 

DISTRICT  RESULTS 
BY: 

n=127 

n=83 

n=157 

n=350 

•     Grade  level 

97% 

93% 

94% 

94% 

•     Curriculum  area 

95% 

93% 

93% 

95% 

•     Sub-scores  within 
curriculum  areas 

76% 

70% 

69% 

50% 

•     Race,  ethnicity 

13% 

8% 

10% 

3% 

•     Gender 

14% 

10% 

10% 

3% 

•     School 

66% 

77% 

76% 

61% 

•     Other 

14% 

12% 

6% 

1% 

SCHOOL  RESULTS 
BY: 

n=126 

n=84 

n=162 

n=376 

•     Grade  level 

96% 

93% 

95% 

90% 

•     Curriculum  area 

97% 

98% 

98% 

95% 

•     Sub-scores  within 
curriculum  areas 

87% 

81% 

87% 

70% 

•     Test  item 

83% 

87% 

85% 

80% 

•     Race,  ethnicity 

13% 

7% 

8% 

5% 

•     Gender 

17% 

11% 

11% 

3% 

•     Feeder  school 

14% 

26% 

15% 

9% 

•     Other 

14% 

6% 

7% 

0% 

Some  respondents  did  no 


answer  all  questions.  Therefore  the  n's  are  slightly  variable. 


Finding  #5;  Teachers  generally  receive  MCAS  results  of  students  who  were  in 
their  school  when  tested,  but  less  often  the  results  of  students  they  actually 
taught  during  the  year  of  testing  or  those  of  students  they  currently  teach. 

Over  90%  of  the  respondents  in  all  groups  indicated  that  teachers  received  results  of 
students  who  were  in  their  schools  when  tested.  However,  the  responses  were  much 
lower  (22%  of  teachers  and  32%  of  principals)  when  respondents  were  asked  if 
teachers  received  results  of  students  who  are  in  their  schools  now  but  were  in  feeder 
schools  when  tested.  Roughly  three-quarters  of  all  groups  indicated  that  the  teachers 
received  results  of  students  they  taught  during  the  year  of  testing.  Thirty-nine  percent  of 
teachers  and  51%  of  principals  said  that  teachers  received  scores  for  students  they 
currently  teach.    Responses  from  all  groups  pointed  to  item  analysis  by  school  as  the 
most  common,  although  73%  of  the  principals  noted  that  teachers  received  item 
analysis  by  student.  Considerable  discrepancies  appeared  in  this  item  particularly 
between  responses  of  superintendents  and  teachers.  For  example,  60%  of  responding 
superintendents  indicated  that  teachers  received  results  of  students  who  were  in  the 
teachers'  schools  now,  but  in  a  feeder  school  when  tested,  compared  with  22%  of  the 
teachers.     Similar  differences  in  responses  of  these  two  groups  occurred  concerning 
teacher  receipt  of  MCAS  results  for  students  they  currently  teach.  See  Table  12. 
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e  12:  Level  of  Analysis  of 

MCAS  Results  Provided  to  Teachers* 

LEVEL  OF  ANALYSIS 

%  OF 

SUPTS. 

n=127 

%OF 

CURRICULUM 

DIRECTORS 

n=84 

%  OF 
PRINCIPALS 

n=162 

%OF 

TEACHERS 

n=376 

RESULTS  OF 
STUDENTS: 

•     Who  were  in  the 
school  when 
tested 

94% 

94% 

96% 

91% 

•     Who  are  in  the 
school  now,  but 
were  in  a  feeder 
school  when 
tested 

60% 

52% 

32% 

22% 

•     That  the  teachers 
taught  during  the 
year  of  testing 

79% 

70% 

75% 

69% 

•     That  they  currently 
teach 

77% 

56% 

51% 

39% 

ITEM  ANALYSIS  BY: 

•      District 

68% 

68% 

62% 

48% 

•      School 

83% 

85% 

90% 

71% 

•      Class  group 

37% 

27% 

30% 

21% 

•      Student 

65% 

45% 

73% 

50% 

*  Some  respondents  did  not  answer  all  questions.  Therefore  the  n's  are  slightly  variable. 

A.  Positive  Practice  from  the  Field:  The  supervisor  of  guidance  prepared  item  analysis  of 
eighth  grade  MCA 5  test  results  showing  the  detail  of  individual  student  responses. 
Results  of  this  breakdown  were  given  to  the  students'  ninth  grade  teachers.  This  allowed 
the  opportunity  for  the  teacher,  armed  with  this  knowledge, to  work  with  this  student  for 
some  of  the  year.  Department  heads  worked  with  teachers  by  discipline  groups.  They 
further  analyzed  eighth  grade  responses  to  evaluate  and  modify  curriculum  and 
instruction,  while  examining  alignment  to  standards/frameworks.  The  district  provided  a 
staff  development  day  for  teachers  to  analyze  and  discuss  the  data,  as  well  as  follow-up 
workshops  concerning  curriculum  components. 


Finding  #9;  The  usefulness  of  MCAS  results,  particularly  for  teachers  to  use  in 
supporting  their  current  students,  is  limited  by  the  receipt  of  student  results 
several  months  into  the  school  year. 

The  issue  of  the  lateness  of  the  MCAS  results  was  raised  in  virtually  every  focus  group 
discussion  by  superintendents,  principals,  and  teachers  and  often  mentioned  in  the 
questionnaire  responses.  Teachers,  particularly,  feeling  the  pressure  to  provide 
targeted  support  for  their  current  students  in  preparing  for  MCAS,  found  receipt  of 
student  scores  halfway  into  the  school  year  to  be  problematic.  Twelve  percent  of  the 
superintendents  who  were  asked  on  the  questionnaire  what  they  found  most 
challenging  in  terms  of  providing  MCAS  results  in  a  useful  form  to  schools  indicated  the 
lateness  of  the  receipt  of  the  results  from  the  state. 
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Finding  #  10:  Teachers  indicate  that  working  with  MCAS  results  has  an  effect  on 
their  choices  of  instructionat  strategies.  They  find  it  motivating  when:  1)  they 
have  ready  access  to  the  MCAS  data;  2)  they  have  opportunities  to  work  together 
with  colleagues;  and  3)  they  are  encouraged  to  do  this  work  by  school  and 
district  administrators. 

The  teacher  questionnaire  asked  teachers  to  respond  to  a  scaled  question  indicating  to 
what  extent  working  with  MCAS  results  (if  they  had  done  so)  had  an  effect  on  the  choice 
of  instructional  strategies  that  they  use  in  their  own  classrooms.  As  Figure  1  shows 
below,  71  %  responded  with  either  a  4  or  5  on  the  5-point  scale  (with  5=to  a  great 
extent). 


Effect  of  MCAS  Data  Usage  on 
Instructional  Choices 


50% 


5  4  3  2  1 

5=To  a  Great  Extent,  1=  Not  at  All 


Figure  1 :  Extent  to  which  teachers  who  have  worked  with  MCAS  results  believe 
this  work  has  had  an  effect  on  the  choice  of  instructional  strategies  that  they  use 
in  their  own  classrooms,  n  =346  teachers,  mean  response  =  3.90 


Over  50%  of  each  group  surveyed  wrote  that  they  had  received  MCAS  results  through 
in-person  meetings  of  some  sort.  Smaller  percentages  indicated  that  results  were 
routed  through  other  people  such  as  department  heads  or  principals  or  that  they  had 
simply  received  hard  copies. 
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In  focus  groups  of  teachers  the  contrast  was  striking  between  different  settings  (on  the 
secondary  level)  where  the  major  variables  were  messages  concerning  MCAS 
responsibilities  and  access  to  MCAS  results.  Many  teachers  who  participated  in  these 
discussions  were  highly  motivated  and  even  exhilarated  to  work  together  with 
colleagues  to  examine  MCAS  results  and  discuss  and  recommend  changes  in  course 
content  and  sequence  as  well  as  classroom  instructional  and  assessment  strategies. 
These  teachers  felt  that  they  had  been  invited  and  enlisted  to  do  this  work  by  their 
district  and  school  administrators.  The  message  they  heard  was  that  they  had  a  charge 
to  do  this  work,  that  their  efforts  were  critically  important  for  their  students,  and  that 
everyone  in  their  school  and  district  carried  equal  responsibility  for  contributing  to  the 
effort  to  improve  student  learning.  These  professionals  had  ready  access  to  MCAS 
results  as  they  became  available.  In  contrast,  teachers  from  other  districts  shared  very 
different  experiences — extreme  frustration  resulting  from  repeated  requests  for  results 
that  were  ignored  or  refused  by  their  central  office,  coupled  with  administrative  finger- 
pointing  and  public  blaming  of  secondary  teachers  as  the  weak  link  in  the  system. 


FINDINGS:  District-Level  Communication,  Incentives, 
and  Leadership 

Finding  #7 1:  Superintendents  and  other  district  and  school  leaders  are 
communicating  expectations  about  the  use  of  MCAS  data,  with  the  most  common 
messages  being  the  importance  of  making  appropriate  changes  in  curriculum 
and  the  need  to  meet  the  overall  goal  for  district  improvement. 

Ninety-eight  percent  of  the  superintendents  indicated  that  they  had  communicated 
expectations  for  use  of  MCAS  results  to  their  principals,  and  92%  to  their  faculty/staff. 
Over  90%  of  the  curriculum  directors  and  principals  and  73%  of  teachers  noted  that 
their  superintendents  had  communicated  such  expectations  to  them. 

All  groups  were  then  asked  to  share  what  expectations  their  superintendents  had 
communicated  to  them.  The  greatest  percentage  of  each  group  (from  35%  to  46%)  fell 
in  the  area  of  making  changes  in  the  curriculum,  particularly  alignment  with  curriculum 
frameworks.  Other  common  responses  included  meeting  the  overall  goal  for  district 
improvement  (31%  of  teachers  and  30%  of  principals)  and  using  data  analysis  to 
identify  strengths  and  weaknesses  in  student  achievement  (22%  of  curriculum  directors, 
17%  of  principals,  and  5%  of  teachers).  Other  expectations  mentioned  to  a  lesser 
degree  included:  providing/participating  in  professional  development  opportunities; 
providing  MCAS  support  for  students  in  need;  making  changes  in  instruction, 
assessment,  and  allocation  of  resources;  and  focusing  efforts  on  test-taking  skills. 

Superintendents,  curriculum  directors,  and  principals  were  asked  what  expectations 
they  communicated  to  teachers  and  teachers  what  expectations  they  heard  from  their 
principal.  Once  again,  the  loudest  message  seems  to  have  been  to  make  curriculum 
changes,  aligning  school/district  curriculum  to  the  frameworks  (36%  of  superintendents, 
38%  of  curriculum  directors,  and  35%  of  principals).  The  largest  percentage  of  teachers 
(24%)  also  indicated  that  this  was  an  expectation  communicated  by  their  principals. 
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In  separate  focus  group  discussions  of  superintendents,  principals,  and  teachers,  the 
messages  that  were  deemed  most  effective  by  many  participants  were: 

•  that  MCAS  results  are  an  important  and  useful  tool  to  improve  student  learning; 

•  that  MCAS  is  an  important  part  of  the  picture  of  student  achievement,  but  not  the 
only  piece;  and 

•  that  the  responsibility  for  improvement  of  student  achievement  (MCAS  being  one 
measure)  lies  with  every  educator  in  the  district. 

Some  focus  group  participants  also  strongly  emphasized  the  responsibility  that  parents 
share. 

Finding  012:  Although  incentives  for  principais  and  teachers  to  use  MCAS  results 
to  make  changes  are  not  broadly  recognized,  some  that  were  noted  include 
monetary  incentives,  time  provided  for  curriculum  alignment  and  professional 
development,  and  opportunities  to  see  students  improve  as  changes  are  made. 

Survey  questions  were  asked  about  the  existence  of  incentives  for  principals,  teachers 
and  curriculum  directors  to  use  MCAS  results  to  make  changes  in  curriculum, 
instruction,  and  other  areas.  Answers  were  fairly  consistent  from  across  groups,  all  with 
less  than  30%  saying  that  incentives  do  exist. 

When  asked  the  nature  of  these  incentives,  37%  of  the  superintendents  who  said  there 
were  incentives  (n=35)  mentioned  money  incentives  for  administrators;  24%  of  the 
principals,  33%  of  the  curriculum  directors,  and  24%  of  the  teachers  indicated  that 
money  incentives  are  in  place  for  teachers  as  well.  These  include  stipends,  raises, 
workshop  fees,  and  pay  for  data  analysis,  curriculum  alignment,  and  working  on  teams 
to  address  issues.  Superintendents  and  principals  noted  that  the  expectation  for  using 
MCAS  is  a  part  of  professional  job  performance.  Superintendents  also  noted  that  the 
publishing  of  scores  and  competitiveness  among  schools  serve  as  incentives  for  others. 
Principals,  curriculum  directors,  and  teachers  all  noted  the  intrinsic  reward  of  seeing 
students  improve.  Principals  noted  time  as  an  incentive — time  for  teachers  to  work 
together,  attend  inservice  and  other  workshops.  Teachers  valued  release  time, 
inservice  time,  and  time  to  work  with  colleagues.  Numerous  opportunities  for 
professional  development  were  noted  by  teachers,  curriculum  directors,  and  principals. 
Teachers  also  valued  the  provision  of  substitutes,  additional  support  from  lead  teachers, 
new  materials,  and  time  to  work  on  curriculum  alignment. 

Most  focus  group  participants  found  it  difficult  to  think  of  existing  incentives,  although 
several  superintendents  and  principals  noted  administrative  contract  agreements  that 
linked  MCAS  improvement  to  school  administrator  salary  increases. 
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Finding  #13:  Educators  surveyed  take  action  in  a  variety  of  leadership  roles  in 
order  to  promote  positive,  responsible  use  of  MCAS  data  for  purposes  of 
improving  schools  and  increasing  achievement  for  all  students. 

Questionnaires  asked  all  respondents  to  describe  important  actions  they  took  as 
leaders  in  order  to  promote  positive,  responsible  use  of  MCAS  data  for  purposes  of 
improving  schools  and  increasing  achievement  for  all  students.  The  superintendents 
most  commonly  described  leadership  actions  of  a  broad  nature,  including:  providing  a 
clear  statement  of  the  importance  of  MCAS  (31%);  providing  support  and  resources  for 
MCAS  improvement  activities,  including  curriculum  work  and  data  analysis  (19%);  and 
providing  and  communicating  about  MCAS  results  to  staff  and  parents  (18%). 

Actions  indicated  by  curriculum  directors  lay  mainly  in  the  areas  of  providing  support 
and  resources  for  MCAS  improvement  activities,  including  curriculum  work  and  data 
analysis  (47%)  and  providing  resources  and  support  for  professional  development 
(22%). 

Principals  focused  mainly  on  providing  support  and  resources  for  MCAS  improvement 
activities,  including  curriculum  work  and  data  analysis  (38%),  and,  to  a  lesser  extent,  on 
providing  a  clear  statement  of  the  importance  of  MCAS  (10%)  and  communicating  about 
and  supporting  standards-based  teaching  (10%). 

Leadership  actions  stated  by  teachers  were  more  broadly  distributed  in  the  areas  of 
focusing  on  open  response  questions,  writing,  and  test-taking  strategies  with  students 
(14%);  sharing  ideas,  strategies,  resources  with  other  teachers  (12%);  providing  support 
and  resources  for  MCAS  improvement  activities,  including  curriculum  work  and  data 
analysis  (11%);  communicating  and  supporting  standards-based  teaching  (9%);  and 
creating  lessons,  tools,  new  curriculum,  MCAS  support  programs  (9%).  Table  13  below 
shows  the  most  frequent  responses  provided  to  this  open-ended  question  by  each 
group. 


Prepared  for  the  Massachusetts  Education  Reform  Review  Commission  by  Learning  Innovations  at  WestEd,  Stoneham,  MA 


V 


HOW  MASSACHUSETTS  SCHOOLS  ARE  USING  MCAS  TO  CHANGE 
CURRICULUM,  INSTRUCTION,  ASSESSMENT,  AND  RESOURCE  ALLOCATION 


23 


Table  13:  Most  Frequently  Cited  Leadership  Actions 


ACTIONS  TAKEN  AS 
A  LEADER 

%  OF 

SUPTS. 

n=121 

%OF 

CURRICULUM 

DIRECTORS 

n=78 

%  OF 

PRINCIPALS 

n=144 

%  OF 
TEACHERS 

n=285 

Providing  a  clear 
statement  of  the 
Importance  of  MCAS 

31% 

3% 

10% 

Providing  support 
and  resources  for 
MCAS  improvement 
activities,  including 
curriculum  work  and 
data  analysis 

19% 

47% 

38% 

11% 

Providing  and 
communicating  about 
MCAS  results  to  staff, 
parents 

18% 

13% 

Communicating  and 
supporting  standards- 
based  teaching 

2% 

3% 

10% 

9% 

Providing  resources 
and  support  for 
professional 
development 

12% 

22% 

4% 

6% 

Creating  lessons, 
tools,  new  curriculum, 
MCAS  support 
programs 

9% 

Focusing  on  open 
response  questions, 
writing,  and  test- 
taking  strategies  with 
students 

14% 

Sharing  ideas, 
strategies,  resources 
with  other  teachers 

12% 

FINDINGS:  Data  Use  Processes 

Finding  #14:  School  administrators  most  commoniy  participate  in  district-wide 
workshops  and  cross-school  administrator  working  groups  in  order  to  learn  from 
MCAS  results  and  use  them  effectively.  Teachers  most  commonly  participate  in 
school-based  grade  level  or  subject  area  working  groups.  These  working  groups 
collaborate  to  make  changes  in  curriculum  and  other  areas,  and  also  contribute 
to  improved  communication  among  teachers  and  increased  buy-in  to  school  and 
district  improvement  efforts. 
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Almost  70%  of  the  superintendents  indicated  that  administrators  were  provided  with 
district-wide  workshops  to  learn  about  how  to  analyze  data.  Sixty-two  percent  indicated 
that  district-wide  workshops  for  administrators  were  also  provided  to  help  them  learn 
from  MCAS  results  and  how  to  use  them  effectively. 

The  same  kinds  of  questions  (in  checklist  form)  were  asked  of  superintendents, 
curriculum  directors,  principals  and  teachers,  concerning  the  types  of  learning 
opportunities  made  available  to  teachers.  Responses  were  fairly  consistent  across 
groups,  with  between  50%  and  78%  of  respondents  from  all  groups  indicating  district- 
wide  and  school-based  workshops.  The  most  prominent  method  for  engaging  teachers 
to  work  with  their  own  MCAS  results  and  use  them  effectively  was  school-based  grade 
level  or  subject  area  working  groups  indicated  by  90%  of  superintendents,  87%  of 
curriculum  directors,  91%  of  principals,  and  76%  of  teachers. 

A  Positive  Practice  from  the  Field:  AW  staff  and  administrators  attended  ahqlf  day  data 
analysis  workshop,  "Making  Data  Work  for  You,"  facilitated  by  a  consultant.  Art,  music, 
health  and  physical  education  teachers  also  attended  to  reduce  fragmentation  in 
accountability.  Teams  of  teachers  from  different  grade  levels  reviewed  MCAS  results  and 
discussed  student  strengths  and  weaknesses  identified.  They  then  generated  strategies 
to  address  these  gaps.  Each  team  reviewed  a  separate  section  of  data  and  reported  out  to 
the  whole  group,  establishing  a  shared  understanding  and  sense  of  accountability.  The 
workshop  assessment  included  the  opportunity  for  teachers  to  outline  their  own  next 
steps.  Teachers  reported  that  the  greatest  value  in  this  experience  lay  in  the  opportunity 
to  share  ideas  and  to  discuss  impact  and  expectations  of  MCAS  adjustments  in  their  own 
classrooms. 

Focus  groups  yielded  many  examples  of  collaborative  efforts  among  teachers  such  as 
district-wide  vertical  teams  by  subject  area;  district-wide  grade  level  teams;  and  school- 
wide  vertical  teams  by  subject  area.  Examples  of  results  of  these  included:  1)  middle 
school  science  teachers  across  grades-revision  of  middle  school  science  content  and 
sequence  to  include  a  content  area  in  which  students  scored  poorly  and  which  was  not 
covered  at  any  grade  level;  and  2)  elementary  teachers  of  language  arts  (across 
grades) — revision  of  curriculum  to  consistently  include  writing  process  and  graphic 
organizers  across  grades. 

Teachers  in  focus  groups  spoke  very  highly  of  their  experiences  in  reviewing  together 
MCAS  results  including  written  compositions  and  in  aligning  curriculum  to  the 
frameworks  based  on  these  results.  For  some  teachers,  particularly  those  at  the  high 
school  level,  this  sort  of  collaborative  work  was  new.  Many  who  participated  in  cross- 
grade  groups  were  excited  to  learn  what  teachers  at  other  grade  levels  were  doing  with 
their  students.  They  greatly  valued  the  collegial  discussions,  sharing,  and  problem 
solving  and  felt  that  this  experience  allowed  them  to  make  significant  contributions 
toward  a  focused  goal  for  improved  student  achievement. 
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On  the  written  questionnaires  all  groups  were  asked  to  respond  to  a  five-point  scaled 
item,  'To  what  extent  do  you  believe  that  working  with  MCAS  results  is  serving  as  a 
positive  stimulus  for  educators  in  your  district  to  work  together  toward  common 
purposes  based  on  identified  student  needs,"  with  (5)  meaning  to  a  great  extent,  and  (1) 
meaning  not  at  all.    On  this  scaled  item,  62%  of  the  superintendents  gave  a  rating  of  (4) 
or  (5);  50%  of  curriculum  directors;  54%  of  principals;  and  40%  of  teachers.  See  Figure 
2  below. 


Working  with  MCAS  Results  Serving  as  a 
Positive  Stimulus  for  Educators 


5  or  4 


2or1 


5=To  a  great  extent,  1=Not  al  all 


QSupts. 
■  Curr.Dirs. 
E3  Princs. 
□  Teachers 


Figure  2:  Extent  to  which  respondents  believe  that  working  with  MCAS  results  is 
serving  as  a  positive  stimulus  for  educators  in  their  districts  to  work  together 
toward  common  purposes  based  on  identified  student  needs,  n  =  130 
superintendents  (mean  of  3.80);  n  =  84  curriculum  directors  (mean  of  3.51);  n  = 
165  principals  (mean  of  3.52);  n  =  372  teachers  (mean  of  3.16). 
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A  Positive  Practice  from  the  Field:  Cross-district  grade  level  teams  came  together  with 
the  Director  of  Curriculum  and  Instruction  in  four  half-day  sessions  with  an  overall  focus 
on  standards-based  teaching.  During  the  first  session  teams  examined  and  compared 
current  curriculum  with  frameworks;  scrutinized  AACAS  questions  and  student  responses; 
and  unveiled  student  weaknesses  and  associated  gaps  in  the  curriculum.  The  second  session 
focused  on  standards-based  instruction,  curriculum  mapping,  and  examining  specif ic 
content— in  this  case,  science.  During  the  third  session  grade  level  teams  mapped  out 
science  units  for  their  grade  levels.  The  fourth  session  included  extensive  opportunities 
for  teachers  to  examine  student  work,  using  their  own  samples  and  discussing  what 
constitutes  excellence.  Teachers  highly  valued  the  opportunity  for  sharing,  collaborative 
planning,  and  general  collegiality.   ^_^  


A  Positive  Practice  from  the  Field:  Vertical  teams  of  teachers  from  each  content  area  in 
grades  1-6  gathered  to  scrutinize  AACAS  questions  and  to  do  a  complete  error  analysis  on 
their  students'  results.  They  then  generalized  patterns  of  errors,  finding  many  in  the  area 
of  English/language  arts.  (Several  incorrect  questions  related  to  poetry  and  poetry 
analysis.)  Teachers  discussed  expectations  for  what  students  would  need  to  know  and 
which  skills  needed  to  be  taught  at  which  level  in  order  to  be  prepared  for  the  fourth 
grade  MCAS  tests.  It  was  felt  that  a  more  coherent  view  of  the  building  of  learning  for 
children  resulted,  and  teachers  had  a  shared  understanding  of  which  skills  to  teach  at 
each  grade  level.  Further  analysis  led  to  realigning  curriculum  and  ended  some  duplication 
of  effort. 
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A  Positive  Practice  from  the  Field— A  District- Wide  Approach 
Data  Analysis: 

•  The  assistant  superintendent  of  a  small,  suburban  district  analyzed  MCAS  data, 
compiling  detailed  results  for  every  school  which  included:  question  and  student 
response  analysis;  analysis  of  strengths  and  weaknesses  in  items  type  such  as  open 
response  and  multiple  choice;  analysis  of  gaps  in  content  areas.  This  person  then 
developed  and  distributed  guidelines  and  a  timeline  for  schools  to  write  an  MCAS 
action  plan. 

•  MCAS  Action  Teams  were  created  at  each  elementary  school  These  included  teachers 
from  every  grade  level;  as  well  as  elementary  specialists  in  art/ music,  physical 
education  and  other  areas. 

•  These  teams  examined  results,  noting  strengths?  and  weaknesses  of  student 
achievement,  and  then  compared  these  to  the  high  benchmarks/standards  already  in 
place  as  district  expectations.  Release  time  was  provided  for  this  team  activity. 

Findings: 

•  Some  specif  ic  problem  areas  noted  were:  grammar  skills  and  application  to  literature; 
vocabulary;  syllabication  skills;  reading  comprehension;  and  writing  organization. 

•  It  was  agreed  that  teachers  needed  training  in  teaching  writing,  time  in  faculty 
meetings  to  share  best  strategies;  and  grade  level  meetings  with  the  reading  resource 
teacher  and  the  media  specialist. 

Actions: 

•  Every  school  developed  an  MCAS  Action  Plan  which  was  to  include: 

•  Subject  areas  addressed 

•  Problem  areas  identified 

•  Intervention  strategies  planned 

•  Professional  development  recommendations 

•  Measurable  goals  for  improvement 

•  Teams  submitted  Action  Plans  to  the  assistant  superintendent  and  ail  teams  reported 
out  a  at  district-wide  meeting. 

•  A  K-12  curriculum  council,  meeting  monthly:  reviewed  MCAS  results;  did  a  full 
curriculum  alignment  check;  analyzed  from  the  district  perspective  what  individual 
schools  across  the  district  need;  and  made  district-wide  recommendations,  with  special 
attention  to  bridge  points  of  grades  5-6  and  grades  8-9. 

Initial  Results: 

•  Strengths/weaknesses  identified  on  the  school  level  as  well  as  district- wide; 

•  Teachers  all  had  a  clearly  delineated,  public  plan,  with  goals,  strategies,  timeline,  and 
measurab 
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A  Positive  Practice  from  the  Field— A  District-Wide  Approach 

Data  Analysis: 

•     The  assistant  superintendent,  responsible  for  MCAS  data  analysis  in  a  large,  urban 

district,  worked  with  staff  to  create  a  district-wide  Assessment  Center,  employing 

three  classroom  teachers  full  time.  These  teachers  analyze  data,  support  curriculum 

coordinators  to  develop/revise  curriculum,  and  provide  student  achievement  data  and 

support  in  using  the  data  to  teachers  in  schools. 

The  database  constructed  for  the  Assessment  Center  provides  student  results  for 

grades  8, 9,  and  10,  including  MCAS,  IOWA  tests,  and  classroom  grades. 

MCAS  tests  are  scrutinized  for  content  areas  and  individual  components,  eg.,  literary 

elements  in  Language  Arts. 

Student  results  are  analyzed  for  curriculum  gaps,  as  well  as  student  strengths  and 

weaknesses. 

Expansion  to  services  and  database  for  primary  and  middle  school  teachers  is  planned 

during  the  2000-2001  school  year. 
Findings: 
Initial  findings  included  needs  identified  for  curriculum  alignment.  For  example: 

Analysis  of  released  MCAS  test  items  revealed  questions  covering  literary  elements 

not  being  covered  in  current  coursework. 

Lower  level  mathematics  students  were  unprepared  for  the  algebra  sections  on  MCAS. 
Actions: 

Teachers  receive  all  available  data  on  their  students  in  September.  The  class  data 

breakdowns  are  accompanied  by  suggestions  for  changes  in  curriculum  coverage  and 

other  practice,  e.g.,  greater  emphasis  on  literary  elements. 

Teachers  participate  in  training  in  data  use.  The  training  emphasizes  positive  use  of 

data  through  reviewing  class  needs  and  choosing  appropriate  materials  that  focus  on 

high  standards. 

Teachers  are  encouraged  to  design  assessments  with  MCAS  type  questions,  e.g.,  more 

open  response  questions  rather  than  emphasis  on  details/facts. 

Problem  of  the  Week  created  at  all  levels  to  address  open-ended  questions  in 

mathematics. 

Block  scheduling  extended  to  full  year  to  address  the  needs  of  lower  level  math 

students. 

Ongoing  groups  are  working  to  align  all  curricula  to  frameworks. 

Benchmarks  are  being  created  for  continual  student  assessment. 
Initial  Results:  Data  analysis  ensured  that: 

Teachers  have  detailed  information  on  incoming  eighth  graders  and  know  how  to  use 

the  data  effectively. 

High  risk  students  are  identified  for  remedial  help. 

Specific  content  areas  identified  as  problem  areas  are  included  in  MCAS  test 

preparation  courses  for  9th  and  10th  graders. 

Curriculum  alignment  efforts  are  informed  by  MCAS  results.  : 
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FINDINGS:  Challenges  Facing  Schools  and  Districts 

Finding  015:  The  greatest  challenge  for  ail  educator  groups  was  in  the  area  of 
managing  change,  ranging  from  educating  parents  and  the  community,  to 
convincing  teachers  and  students,  to  maintaining  balance  and  a  positive 
atmosphere,  to  managing  to  cover  required  curriculum.  Lack  of  time  was  a 
common  theme  from  all  four  groups  as  well. 

All  four  constituencies  surveyed  were  asked  to  describe  (In  open-ended  questions)  their 
greatest  challenge  in  promoting  positive,  responsible  use  of  MCAS  data  for  purposes  of 
improving  schools  and  increasing  achievement  for  all  students.  The  most  frequently 
cited  responses  are  presented  in  Table  14  below. 

Table  14:  Most  Frequently  Cited  Challenges  Facing  Schools  and  Districts 


GREATEST 
CHALLENGES 

%  OF 

SUPTS. 

n=117 

%OF 

CURRICULUM 

DIRECTORS 

n=78 

%  OF 

PRINCIPALS 

n=147 

■   ■■ 

%OF 

TEACHERS 

n=322 

Managing  change, 
e.g.,  getting  everyone 
on  board,  maintaining 
positive  atmosphere 

36% 

30% 

36% 

25% 

Lack  of  time 

14% 

9% 

14% 

17% 

Dealing  with  negative 
external  messages 

23% 

3% 

2% 

Issues  with  MCAS 
results  data, 
especially  late  receipt 

17% 

9% 

3% 

7% 

Concern  for  students 

4% 

6% 

11% 

20% 

•    The  greatest  percentages  of  all  groups  (ranging  from  25%  of  teachers  to  36%  of 
superintendents)  indicated  challenges  that  can  be  described  as  managing  change 
in  their  domains:  getting  everyone  on  board;  ensuring  accountability  of  all;  educating 
parents  and  community  members;  convincing  teachers  and  students  of  the  value  of 
MCAS;  maintaining  balance,  momentum,  and  positive  atmosphere;  dealing  with 
student  and  parent  apathy;  and  covering  an  expanding  curriculum. 

Sample  Comments  about  Managing  Change: 

"Changing  the  perception  of  teachers  that  instruction/assessment  must  change. 
Helping  teachers  to  understand  the  connection  between  MCAS  and  student 
achievement.  Getting  teachers  to  understand  and  accept  the  importance  of 
using  data. "  -  a  superintendent 
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"My  greatest  challenge  is  to  create  a  proactive  rather  than  reactive  response  to 
data.  Teachers  are  stressed  and  overwhelmed. "  -  a  curriculum  director 

"With  over  300  teachers  in  [high  school]  faculty,  my  greatest  challenge  as 
principal  is  to  enlist  all  of  them  (not  just  the  ones  in  testing  areas)  in  the  MCAS 
preparation  process.  Everyone  must  view  this  as  a  school-wide  goal." 

-  a  principal 

"Convincing  students  how  important  MCAS  is  in  terms  of  always  trying  to  do 
their  best  and  focus  on  learning. "  -  a  principal 

"My  department  and  I  are  overwhelmed  by  the  amount  of  content  which  needs 
to  be  covered  in  each  course.  Some  students  score  low  because  the  material 
has  not  yet  been  taught. "  -a  teacher 


Other  challenges  raised  were  lack  of  time  (14%  of  superintendents  and  principals, 
and  17%  of  teachers); 

Sample  Comments  about  Lack  of  Time: 

"Unquestionably — TIME!  Teachers  need  time  to  do  this  work  which  they  are 
willing  to  do  and  want  to  do. "  -  a  superintendent 

"Finding  time  to  work  with  teachers.  Teachers  are  at  times  too  overwhelmed 
with  the  everyday  functioning  of  their  classrooms  to  worry  about  what  the 
MCAS  may  tell  them. "  --  a  curriculum  director 

"  Time!  Finding  the  time  for  all  faculty  and  staff  to  come  together  to  brainstorm 
ideas,  implement  change,  and  achieve  success. "  -  a  principal 

"Finding  time  to  collaborate  with  peers  and  share  ideas  and  strategies,  and 
attempting  to  convince  students  that  they  can  and  will  succeed. "   -  a  teacher 


•    Negative  external  messages  from  the  media,  community,  and  political  arena  (23% 
of  superintendents); 

Sample  Comments  about  Negative  External  Messages: 

"Keeping  great  teachers  motivated  while  the  press  and  government  and  the 
DOE  continue  to  whip  teachers  and  the  education  profession." 

-  a  superintendent 

"Negative  media  attention;  public  criticism  by  the  school  committee." 

-  a  principal 
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Issues  with  the  MCAS  results  such  as  misuse  and  late  arrival  (17%  of 
superintendents); 

Sample  Comments  about  Issues  about  MCAS  Data: 

'The  lack  of  timely  return  of  the  test  scores  presents  a  major  challenge.  When 
the  MCAS  data  arrives  in  December,  principals  and  curriculum  leaders  must 
readjust  goals  that  had  been  set  for  the  school  year  to  reflect  the  new 
information  for  MCAS.  This  process  means  that  valuable  time  was  lost  that 
could  have  been  targeted  to  specific  students  and  specific  areas  of  weakness  in 
the  program. "  -  a  superintendent 

'The  quality  of  this  year's  data  was  so  poor  that  we  had  to  redevelop  the 
database  to  make  it  consistent  with  last  year's  data.  The  information  comes  too 
late."  -  a  superintendent 

"As  an  intermediate  teacher  in  the  elementary  school,  I  see  a  gap  in  sharing 
results  from  grade  8  MCAS  since  it  is  administered  in  the  junior  high.  Efforts 
have  begun  to  encourage  dialogue  vertically  in  the  elementary  schools, 
however,  since  we  are  in  separate  buildings,  it  is  more  difficult  for  grades  5-8  to 
share."  -a  teacher 

"Receiving  data  in  November,  December  of  the  following  school  year  on 
students  I  will  not  see  again. "  —  a  teacher 

"The  biggest  challenge  is  trying  to  make  use  of  results  that  are  reported  too  late 
to  be  useful  and  to  feel  confident  about  results  that  are  not  representative  of 
student  knowledge/ability. "  -  a  teacher 


and  concern  for  students  (20%  of  teachers). 

Sample  Comments  about  Concern  for  Students: 

"Identifying  weaknesses  and  strengths  of  students  and  initiating  programs  for  at 
risk  students  to  meet  their  needs. "  ~  a  principal 

"One  area  of  concern  continues  to  be  the  SPED  [special  education]  population. 
Two  years  of  results  have  shown  that  this  group  is  struggling  with  the  Language 
Arts  portion  of  the  test.  Developing  new  instructional  techniques  in  order  to 
boost  this  population  has  been  very  difficult. "  -a  teacher 
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Some  of  the  most  commonly  cited  challenges  raised  in  focus  groups  included: 

•  retaining  good  teachers  and  principals  in  low-performing  schools; 

•  helping  students  reduce  test-taking  anxiety  and  avoiding  increases  in  student 
dropout; 

•  maintaining  valuable  educational  experiences  that  are  not  directly  tested  on  the 
MCAS,  e.g.,  vocational  studies,  the  arts,  and  school-to-career  activities; 

•  dealing  with  increased  negative  competition  among  schools  within  a  district  or 
among  neighboring  districts; 

•  realigning  curriculum  and  making  appropriate  decisions  about  acquisition  of 
textbooks  and  other  materials  when  the  state  curriculum  frameworks  continue  to 
undergo  significant  change;  and 

•  meeting  snowballing  requirements  of  time  and  personnel  to  provide  support  and 
remediation  for  students  who  do  not  fare  well  on  the  MCAS. 

Superintendents  were  asked  on  the  questionnaire  to  share  what  they  found  most 
challenging  In  terms  of  providing  MCAS  results  in  a  useful  form  to  schools.  The 
difficulties  cited  by  the  greatest  percent  of  superintendents  were  the  actual  data  analysis 
(20%  of  the  respondents),  the  time  required  (19%),  lateness  of  the  receipt  of  the  results 
from  the  state  (12%),  and  problems  encountered  using  the  Department  of  Education 
data  CD. 


REFLECTIONS 

Based  on  the  findings  of  this  study,  educators  in  the  state  are  certainly  making  strides 
toward  using  MCAS  in  the  way  it  was  designed — to  serve  as  a  tool  to  support  efforts  to 
improve  student  learning  by  providing  a  consistent  measure  of  achievement  linked  to 
the  statewide  curriculum  frameworks.    Clearly,  in  the  districts  that  participated  in  this 
study  (that  were  regionally  representative  as  well  as  representative  of  the  variety  of 
types  of  communities  identified  by  the  Massachusetts  Department  of  Education),  school 
and  district  administration  and  staff  are  analyzing,  distributing,  and  studying  student 
achievement  data  in  ways  and  toward  purposes  that  were  rare  only  a  few  years  ago. 
Based  on  this  work,  they  are  also  making  significant  changes  in  the  areas  of  curriculum, 
instruction,  assessment,  and  allocation  of  resources.  Perhaps  most  importantly,  MCAS 
results  are  reaching  teachers — the  change-makers  in  the  classroom — and  teachers  are 
using  the  data  in  important  ways.  Over  70%  of  the  teachers  responding  to  a  scaled  item 
on  the  questionnaire  indicated  that  that  their  use  of  MCAS  data  has  had  an  effect  on  the 
instructional  strategies  they  choose  to  use  in  their  classrooms.  The  degree  to  which  this 
sort  of  activity  is  taking  place  in  non-participating  districts  is,  of  course,  unknown. 
Nevertheless,  even  though  it  is  still  too  early  to  draw  many  conclusions  about  the  impact 
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of  this  work  on  student  learning,  clearly  these  important  efforts  are  moving  in  a  positive 
direction. 

In  this  new  direction: 

•  A  broad  range  of  educators  is  taking  responsibility  and  ownership  of  the  task  of 
deeply  examining  student  achievement  data.  It  is  no  longer  just  the  assistant 
superintendent  or  a  few  others  who  look  carefully  for  patterns,  trends,  and  reasons 
for  student  achievement  results  being  what  they  are.  It  is  becoming  common 
practice  for  curriculum  leaders,  principals,  and  teachers  to  take  on  this  responsibility, 
identifying  student  strengths  and  weaknesses,  recognizing  gaps  in  curriculum, 
instruction,  and  assessment,  and  making  the  kinds  of  changes  they  believe  can 
make  a  difference. 

The  results  of  this  study  show  that  district  and  school  MCAS  scores  are  being 
distributed  to  schools  and  to  teachers  and  studied  in  disaggregated  form  by 
curriculum  area,  curriculum  area  sub-scores,  grade  level,  and  by  individual  item.  In 
some  settings  many  additional  analyses  or  breakdowns  are  conducted.  It  has  also 
been  learned  that  access  to  MCAS  results  and  the  timing  of  these  is  critical  for 
optimal  effectiveness  of  use  to  support  students.  Many  teachers  participating  in  this 
study  indicated  that  they  do  not  receive  pertinent  MCAS  data  for  students  they  teach 
currently,  and  often  not  for  students  they  taught  during  a  testing  year. 

It  seems  that  district-level  communication  so  far  has  been  primarily  to  establish  the 
importance  of  using  MCAS  results  effectively,  with  a  major  emphasis  on  using 
results  to  ensure  alignment  of  curriculum  to  frameworks  and  identification  of 
curriculum  gaps,  overlaps,  or  faulty  sequence.  Although  the  point  was  made  again 
and  again  in  focus  groups  that  curriculum  alignment  had  been  underway  for  years,  it 
is  clear  that  the  pressure  of  MCAS  has  provided  the  impetus  for  much  more 
aggressive  focus  on  the  curriculum  that  is  actually  taught.  Further,  teachers  are 
being  asked  to  look  beyond  what  they  teach  in  their  own  classrooms  at  their  own 
levels,  and  to  see  where  the  curriculum  they  teach  fits  into  the  whole  K-12  picture. 

•  Teachers  are  working  together  more  by  grade  level,  subject  area,  and  across 
grades — talking,  sharing,  comparing,  troubleshooting,  taking  responsibility,  and 
more  actively  looking  at  the  impact  of  their  teaching  on  at  least  one  measure  of 
student  achievement. 

Clearly  the  processes  devised  for  engaging  educators  in  studying  and  using  MCAS 
results  are  very  important,  with  the  most  prominent  method  used  for  teachers  being 
school  based  grade  level  or  subject  area  working  groups.  Cross  grade  groupings 
seem  to  be  particularly  effective  in  identifying  curriculum  problems,  as  well  as 
establishing  some  common  instructional  approaches  across  grades. 
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•  Significant  changes  are  being  made  in  the  areas  of  curriculum,  instruction, 
assessment,  and  allocation  of  resources. 

In  addition  to  the  activities  and  processes  that  have  been  created  for  the  analysis, 
distribution,  and  discussion  of  MCAS  data,  it  is  clear  that  educators  are  making 
decisions  and  changes  in  curriculum  based  on  the  student  achievement  results. 
Consistently,  despite  years  of  resistance  to  providing  algebra  to  all  students  by 
grade  9,  now,  because  of  MCAS  graduation  requirements,  that  curriculum  change  is 
actually  happening.  Identified  missing  topics  in  technology,  literature,  social  studies, 
and  science  are  also  being  inserted  into  or  enhanced  within  curricula.  Writing  and 
critical  thinking  skills  are  being  included  across  disciplines. 

Changes  in  instruction  and  assessment  are  generally  considered  much  harder  to 
make  than  curriculum  adjustments,  especially  on  a  large  scale.  A  large  percentage 
of  teachers  surveyed  indicated  that  their  work  in  analysis  of  MCAS  results  has  had 
an  effect  on  their  choices  of  instructional  strategies.  The  most  commonly  cited 
changes  in  this  area  are  increased  writing  instruction  across  the  curriculum,  and 
more  use  of  graphic  organizers  and  rubrics  for  writing  assignments.  Many  teachers 
indicate  more  frequent  use  of  writing  prompts  for  students.  It  seems  like  teachers 
are  spending  a  great  deal  of  energy  on  teaching  test  taking  skills,  and  particularly 
helping  students  respond  to  open  response  questions  and  to  explain  their  thinking 
and  how  they  arrived  at  solutions  to  problems,  particularly  in  mathematics  and 
science.  Such  changes  reflect  classroom  movement  toward  mathematics  standards 
of  the  National  Council  of  Teachers  of  Mathematics  and  toward  the  state  curriculum 
standards  in  mathematics  and  science. 

Resources  are  being  allocated  for  professional  development  and  curriculum  review 
work  of  teachers,  new  instructional  materials,  and  additional  student  support 
services  (focused  on  improving  MCAS  success)  before,  during,  and  after  school. 

•  All  staff  are  being  held  accountable  for  student  achievement,  not  just  those  in  a  few 
disciplines  or  grade  levels. 

•  Educators  throughout  the  state  are  developing  positive  educational  practices  using 
MCAS  results  as  a  basis  -  practices  that  can  be  shared  with  others.  Many  such 
practices  have  emerged  through  this  study  (some  of  them  shared  in  this  report) 
including  instructional  and  assessment  practices  of  individual  teachers,  data  use 
strategies  for  groups  of  educators,  as  well  as  whole  district  approaches  to 
maximizing  the  use  of  MCAS  in  support  of  student  learning. 

However,  some  questions  arise  as  educators  in  the  state  move  in  this  new  direction: 

•  Can  this  analysis  of  student  achievement  data  become  a  consistent  and  important 
part  of  the  culture  of  schools  and  districts?  Will  it  become  a  way  of  working  and 
troubleshooting  that  will  be  continued,  built  upon,  and  deepened? 
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Will  efforts  continue  so  that  more  and  more  teachers  and  administrators,  especially 
new  arrivals  to  the  scene,  become  adept  and  comfortable  with  examining  student 
scores  and  student  work  effectively? 

Once  significant  changes  are  made  in  curriculum,  instruction,  and  assessment,  will 
results  be  revisited  year  after  year  to  determine  the  impact  of  these  changes  over 
time  within  a  range  of  settings,  disciplines,  and  learners? 

Will  changes  go  beyond  curriculum  as  it  is  written  and  reach  the  level  of  curriculum 
that  is  taught? 

Will  educators  continue  to  probe  and  question  their  instructional  and  assessment 
practices  and  make  changes  that  go  well  beyond  helping  students  with  test-taking 
skills  and  writing  process? 

Will  school  and  district  resources  continue  to  be  invested  in  critical  professional 
development  and  curriculum  review  activities  that  are  so  vital  for  educators  in  our 
schools? 


RECOMMENDATIONS 

It  is  most  clear  from  this  study  that  timing,  communication,  attitude,  and  process  matter 
if  MCAS  is  to  achieve  its  purpose  within  Massachusetts  education  reform.  Given  these 
and  other  findings  of  this  study — both  encouraging  and  challenging — we  offer  the 
following  recommendations: 

To  the  Department  of  Education 

•  Make  it  a  priority  to  provide  MCAS  results  to  districts  before  the  beginning  of  each 
school  year,  as  opposed  to  November  or  December.  A  great  deal  of  momentum  and 
effectiveness  of  use  is  lost  by  not  having  the  data  available  in  a  timely  manner. 

•  Listen  and  respond  to  suggestions  from  the  field,  especially  concerning  the  quality  of 
the  MCAS  result  reports  and  CDs  provided  to  districts. 

•  Support  opportunities  for  districts  and  schools  to  share  and  troubleshoot  their 
approaches  to  using  MCAS  results,  through  in-person  and  on-line  forums  that  are 
non-threatening  and  non-monitoring  in  nature. 

•  Review  district  and  school  utilization  of  MCAS  results  and  actively  disseminate 
identified  positive  practices. 

•  Continue  efforts  to  develop  other  measures  of  student  achievement  to  be  used 
within  education  reform  efforts. 
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To  District  and  School  Educators 

•  Keep  the  beat  going — not  fear  of  MCAS  or  needless  threat  or  competition — but  the 
importance  and  usefulness  of  using  MCAS  results  openly  and  responsibly. 
Encourage  educators  on  all  levels  to  continue  probing  curricula  and  classroom 
practice,  moving  to  higher  levels  of  reflection  and  change. 

•  Set  and  communicate  clear  expectations  and  responsibilities  for  all  within  schools 
and  districts  for  using  MCAS  results  and  other  measures  of  student  achievement  to 
improve  student  learning. 

•  Distribute  the  appropriate  MCAS  results  to  teachers  so  that  they  can  maximize  the 
use  of  these  to  support  and  prepare  their  students.  In  order  for  teachers  to  make 
appropriate  changes  in  curriculum  taught  and  in  instructional  and  assessment 
strategies  used,  it  is  imperative  that  they  receive  the  results  of  the  students  they 
taught  previously  as  well  as  those  of  their  current  students. 

•  Support  and  encourage  expansion  of  collaborative  efforts  in  using  MCAS  results, 
especially  drawing  together  grade  level,  subject  area,  and  cross  grade  groupings  of 
educators. 


• 


Continue  to  develop  staff  leaders  in  all  schools  who  have  confidence  and  expertise 
in  using  data  for  making  decisions  about  curriculum,  instruction,  assessment,  and 
allocation  of  resources.  Support  their  efforts  to  bring  other  staff  members  into  this 
work. 

Reward  and  recognize  the  accomplishments  of  collaborative  efforts.  Celebrate 
improvements  in  student  achievement  that  are  manifested  by  MCAS  results  as  well 
as  other  measures. 

Encourage  development,  use,  and  sharing  of  wider  varieties  of  instructional  and 
assessment  strategies  that  show  promise  in  better  preparing  students  to  meet  the 
rigorous  requirements  of  MCAS  as  well  as  those  of  future  academic  or  work-related 
activities. 

Encourage  the  development  and  use  of  other  measures  of  student  achievement  that 
complement  the  MCAS  data. 

Document  intentional  change  efforts  and  the  shifts  in  student  achievement  that  may 
be  linked  to  those  efforts. 
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APPENDIX  A  :  Blank  Questionnaires  for 

Superintendents,  Curriculum  Directors, 
Principals,  and  Teachers 


Complete  List  of  Districts  from  Whom 
Questionnaires  Were  Received 
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SUPERINTENDENT  QUESTIONNAIRE 

Please  note  that  this  study  is  specifically  focused  on  the  use  of  MCAS  results  in  schools  and  districts, 
rather  than  on  related  activities  or  concerns  such  as  general  preparation  for  testing  or  alignment  with 
curriculum  frameworks.  Thank  you  for  taking  the  time  to  complete  this  important  survey. 

1 .    Please  indicate  (with  a  check  mark)  which  of  the  following  individuals  or  groups  have  been 
responsible  for  analysis  of  MCAS  data  in  your  district  during  the  current  school  year. 
□Superintendent                                          □  Assistant  Superintendent 
□Curriculum  Director(s)                              □  Outside  Consultant 
□Principals                                                   □District-wide  Teams 
□School  Teams                                             □  Grade  Level  Teams 
□Subject  Area  Teams  □Other(s)  (Please  specify.) 


2.  What  computer  software  packages  or  other  data  analysis  tools  (if  any)  are  being  used  in  your  district? 

Software  packages: 

Other  data  analysis  tools: 

3.  Have  the  MCAS  results  from  the  1998-1999  testing  been  disseminated  to  all  schools  in  your  district? 
□YES  QNO 

If  YES,  by  when?  (Month: ,  Year: ) 

If  YES,  please  continue  with  question  #4. 

If  NO,  please  explain._ 

If  NO,  please  move  to  Question  #7. 

4.  To  whom  were  the  MCAS  results  disseminated  in  the  schools  in  your  district? 
□Principals 

□All  faculty/staff 

□Selected  faculty/staff  (please  specify) 


5.    Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  schools  from  your 
district  office. 
□District  results 

Please  indicate  the  types  of  breakdowns  provided  in  district  results: 

□By  grade  level 

□By  curriculum  area 

□By  subscores  within  curriculum  areas 

□By  race,  ethnicity 

□By  gender 

□By  school 

□Others  (Please  specify)        


□School  results 

Please  indicate  the  types  of  breakdowns  provided  in  individual  school  results: 
□By  grade  level 
□By  curriculum  area 
□By  subscores  within  curriculum  areas 
□By  test  item 
□By  race,  ethnicity 
□By  gender 
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□By  feeder  school 

□Others  (Please  specify ) 

Please  indicate  the  students  for  which  MCAS  results  were  sent  to  schools: 
□Results  of  students  who  were  in  the  school  when  tested 

□Results  of  students  who  are  in  that  school  now,  but  were  in  a  feeder  school  when  tested 
□Other  (Please  specify.) 

Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  individual 
teachers  in  your  district's  schools: 

□Results  of  students  they  taught  during  the  year  of  testing 

□Results  of  students  they  currently  teach 

□item  analysis  by  district 

□item  analysis  by  school 

□item  analysis  by  class  group 

□item  analysis  by  student 

□Other  (Please  specify.) 


6.    What  was  the  primary  method  for  dissemination  of  MCAS  results  to  the  schools? 


7.    What  was  the  greatest  challenge  facing  the  district  office  in  providing  MCAS  results  in  a  useful  form 
to  schools? 


8.    Have  you  communicated  with  school  administrators  in  your  district  about  district  expectations  for 
school  use  of  MCAS  results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource 
allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  expectations: 


Does  your  district  provide  any  specific  incentives  for  school  administrators  to  encourage  the  use  of 
MCAS  results  to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  incentives: 


10.  What  professional  development  methods  have  you  used  in  your  district  to  provide  school 
administrators  with  the  knowledge  and  skills  they  need  to  analyze  and  use  MCAS  results 
effectively? 

Q  District-wide  workshops  for  administrators  (Topics/focus: ) 

Q  Cross-school  administrator  working  groups  (Topics/focus: ) 

Q  None 

□  Other  (Please  specify.) 
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11.  What  methods  have  you  used  in  your  district  to  engage  school  administrators  in  learning  from  their 
own  schools'  MCAS  results  and  using  them  effectively? 

U  District-wide  workshops  for  administrators 

Q  Cross-school  administrator  working  groups 

Q  None 

Q  Other  (Please  specify.) 

12.  What  (if  any)  outside  resources,  e.g.,  organizations  or  consultants,  has  your  district  used  to  help  your 
school  administrators  analyze  and  use  their  MCAS  results  effectively? 


13.  Have  you  communicated  with  faculty  and  staff  about  district  expectations  for  the  school  use  of 
MCAS  results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource  allocation? 
□  YES       □  NO 
If  YES,  please  describe  these  expectations: 


14.  Does  your  district  provide  any  specific  incentives  for  faculty  and  staff  to  encourage  the  use  of 
MCAS  results  to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  incentives: 


15.  What  professional  development  methods  have  you  used  in  your  district  to  provide  school  faculty  and 
staff  with  the  knowledge  and  skills  they  need  to  analyze  and  use  MCAS  results  effectively? 

O  District- wide  workshops  for  teachers  (Topics/focus: ) 

LJ  School-based  workshops  for  teachers  (Topics/focus: ) 

U  Other  (Please  specify.) 


16.  What  methods  have  you  used  in  your  district  to  engage  school  faculty  and  staff  in  learning  from 
their  own  schools'  MCAS  results  and  using  them  effectively? 

ul  District-wide  workshops  for  teachers 

U  Cross-district  grade  level  working  groups 

□  School-based  grade  level  or  subject  area  working  groups 

1— )  Vertical  (across  grades)  working  groups  in  subject  areas 

Q  School-based,  cross-grade,  and/or  cross-subject  working  groups 

LJ  Other  (Please  specify.) 

17.  Please  circle  the  strategy/activity  in  the  checklist  in  #15  above  that  you  have  found  to  be  most 
effective. 

Briefly  explain  why  you  believe  this  strategy/activity  to  be  effective: 
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18.  What  (if  any)  outside  resources,  e.g.,  organizations  or  consultants,  has  your  district  used  to  help  your 
school  faculty  and  staff  analyze  and  use  their  MCAS  results  effectively? 


19.  To  what  extent  do  you  believe  that  working  with  MCAS  results  is  serving  as  a  positive  stimulus  for 
educators  in  your  district  to  work  together  toward  common  purposes  based  on  identified  student 
needs?  (Circle  the  number  of  your  response.) 


To  a  Great  Extent 

(5)  (4) 


(3) 


(2) 


Not  at  All 

(1) 


Comments: 


20.  In  thinking  about  your  own  district's  MCAS  results,  please  share  with  us  what  you  believe  to  be  the 
three  most  important  student  achievement  gaps  or  concerns  identified? 

1. 


Given  these  areas  of  need  for  improvement  in  student  achievement,  please  respond  to  the  following 
questions  about  actions  taken  in  the  areas  of  curriculum,  instruction,  assessment,  and  resource 
allocation. 

21.  Have  curriculum  changes  been  made  or  identified  to  address  any  of  these  student  achievement  gaps 
or  concerns?     QYES  QNO 
If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  curriculum  change: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change,  e.g.,  cross-level  team  of  8  English 
Language  Arts  teachers  and  the  curriculum  director? 


What  results  of  this  change,  if  any,  have  been  observed? 


22.  In  these  or  other  curricular  changes  planned  by  your  district,  please  indicate  any  courses  or  particular 
areas  of  your  district's  curricula  that  have  been  eliminated  or  added. 

Courses  eliminated: 


Courses  added: 


Curricular  areas  of  study  eliminated: 
Curricular  areas  of  study  added: 
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23.  Have  instructional  changes  been  made  or  identified  to  address  any  of  these  student  achievement  gaps 
or  concerns?  QYES  QNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  instructional  change: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 


What  results  of  this  change,  if  any,  have  been  observed? 


24.  Have  changes  in  classroom  assessment  been  made  or  identified  to  address  any  of  these  student 
achievement  gaps  or  concerns?         QYES  QNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  classroom 
assessment: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 
What  results  of  this  change,  if  any,  have  been  observed? 


25.  Has  the  allocation  of  resources  in  your  district  changed  due  to  efforts  to  address  any  of  these  student 
achievement  gaps  or  concerns?         QYES  QNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  allocation  of 
resources: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 


What  results  of  this  change,  if  any,  have  been  observed? 
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26.  Please  share  with  us  in  a  few  words  your  greatest  challenge  in  promoting  positive,  responsible  use 
of  MCAS  data  for  purposes  of  improving  schools  and  increasing  achievement  for  all  students. 


27.  What  do  you  believe  is  the  most  important  action  you  have  taken  as  a  leader  in  your  district  to 

promote  positive,  responsible  use  of  MCAS  data  for  purposes  of  improving  schools  and  increasing 
achievement  for  all  students? 


BACKGROUND 

Please  note  that  this  information  will  be  used  for  research  purposes  only  and  will  be  kept  strictly 
confidential.  No  individuals  or  districts  will  be  identified  or  recognizable  in  the  final  report. 

Name  of  district: 


Number  of  students: 


Percent  of  students  classified  as  free  or  reduced  lunch: 


FOLLOW-UP 


LJ         Please  indicate  here  if  there  is  a  particular  strategy  or  practice  in  terms  of  using  MCAS  data  that 
is  being  used  in  your  district  that  you  think  is  particularly  valuable  and  about  which  you  or 
someone  else  in  your  district  would  be  willing  to  be  interviewed  by  telephone  for  this  study. 

Strategy  or  practice: 


Name  of  person  to  be  called: 
Position: 


Telephone  number: 
E-Mail: 


□         Please  indicate  here  if  you  or  someone  else  in  your  district  would  like  to  provide  additional  input 
to  this  research  study  through  a  telephone  interview 

Name: 

Position: 


Telephone  number: 
E-Mail: 


THANK  YOU  FOR  COMPLETING  THIS  QUESTIONNAIRE! 

Please  return  this  questionnaire  in  the  stamped,  self-addressed  envelope  provided  to: 

Learning  Innovations — A  Division  of  WestEd 

91  Montvale  Avenue 

Stoneham,  MA  02180 
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CURRICULUM  DIRECTOR  QUESTIONNAIRE 

Please  note  that  this  study  is  specifically  focused  on  the  use  of  MCAS  results  in  schools  and  districts, 
rather  than  on  related  activities  or  concerns  such  as  general  preparation  for  testing  or  alignment  with 
curriculum  frameworks.  Thank  you  for  taking  the  time  to  complete  this  important  survey. 

1 .    Please  indicate  (with  a  check  mark)  which  of  the  following  individuals  or  groups  have  been 
responsible  for  analysis  of  MCAS  data  in  your  district  during  the  current  school  year. 

□  Superintendent  □  Assistant  Superintendent 
□Curriculum  Director(s)  □  Outside  Consultant 
□Principals  □District- wide  Teams 
□School  Teams  □  Grade  Level  Teams 

□  Subj  ect  Area  Teams  □  Other(s)  (Please  specify.) 


2.  Have  you  personally  been  involved  in  analyzing  MCAS  results?  QYES     □NO 

3.  What  computer  software  packages  or  other  data  analysis  tools  (if  any)  are  being  used  in  your  district? 

Software  packages: 

Other  data  analysis  tools: 

4.  Have  teachers  in  your  curriculum  area  received  MCAS  results? 

□YES    Qno 

If  YES,  by  when?  (Month: ,  Year: ) 

If  YES,  please  continue  with  question  #5. 

If  NO,  please  explain. 

If  NO,  please  move  to  Question  #7. 

5.  Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  teachers  in  your 
curriculum  area. 

□District  results 

Please  indicate  the  types  of  breakdowns  provided  in  district  results: 

□By  grade  level 

□By  curriculum  area 

□By  subscores  within  curriculum  areas 

□By  race,  ethnicity 

□By  gender 

□By  school 

□Others  (Please  specify)     

□School  results 

Please  indicate  the  types  of  breakdowns  provided  in  individual  school  results: 

□By  grade  level 

□By  curriculum  area 

□By  subscores  within  curriculum  areas 

□By  test  item 

□By  race,  ethnicity 

□By  gender 

□By  feeder  school 

□Others  (Please  specify) 
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Please  indicate  the  students  for  which  MCAS  results  were  sent  to  schools: 
□Results  of  students  who  were  in  the  school  when  tested 

□Results  of  students  who  are  in  that  school  now,  but  were  in  a  feeder  school  when  tested 
□Other  (Please  specify.) 

Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  individual 
teachers  in  your  curriculum  area: 

□Results  of  students  they  taught  during  the  year  of  testing 

□Results  of  students  they  currently  teach 

□item  analysis  by  district 

□item  analysis  by  school 

□item  analysis  by  class  group 

□item  analysis  by  student 

□Other  (Please  specify.) 


What  was  the  primary  method  for  dissemination  of  MCAS  results  to  the  teachers  in  your 
curriculum  area? 


7.    Has  your  superintendent  communicated  with  curriculum  directors  about  district  expectations  for  use 
of  MCAS  results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource  allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  expectations: 


8.    Does  your  district  provide  any  specific  incentives  for  curriculum  directors  to  encourage  the  use  of 
MCAS  results  to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 
□  YES       □  NO 
If  YES,  please  describe  these  incentives: 


Have  you  communicated  with  teachers  in  your  curriculum  area  about  district  expectations  for  use 
of  MCAS  results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource  allocation? 
□  YES       □  NO 
If  YES,  please  describe  these  expectations: 


10.  Does  your  district  provide  any  specific  incentives  for  teachers  to  encourage  the  use  of  MCAS  results 
to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 
□  YES       □  NO 
If  YES,  please  describe  these  incentives: 
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11.  What  professional  development  methods  have  been  used  in  your  district  to  provide  teachers  in  your 
curriculum  area  with  the  knowledge  and  skills  they  need  to  analyze  and  use  MCAS  results 
effectively? 

□  District- wide  workshops  for  teachers  (Topics/focus: ) 

□  School-based  workshops  for  teachers  (Topics/focus: ) 

Q  Other  (Please  specify.) 

12.  What  methods  have  been  used  in  your  district  to  engage  teachers  in  your  curriculum  area  in 
learning  from  their  own  schools'  MCAS  results  and  using  them  effectively? 

Li  District-wide  workshops  for  teachers 

Q  Cross-district  grade  level  working  groups 

LJ  School-based  grade  level  or  subject  area  working  groups 

LI  Vertical  (across  grades)  working  groups  in  subject  areas 

L I  School-based,  cross-grade,  and/or  cross-subject  working  groups 

LJ  Other  (Please  specify.) 

13.  Please  circle  the  strategy/activity  in  the  checklist  in  #12  above  that  you  have  found  to  be  most 
effective. 

Briefly  explain  why  you  believe  this  strategy/activity  to  be  effective: 


14.  What  (if  any)  outside  resources,  e.g.,  organizations  or  consultants,  has  your  district  used  to  help 
teachers  in  your  curriculum  area  to  analyze  and  use  their  MCAS  results  effectively? 


15.  To  what  extent  do  you  believe  that  working  with  MCAS  results  is  serving  as  a  positive  stimulus  for 
teachers  in  your  curriculum  area  to  work  together  toward  common  purposes  based  on  identified 
student  needs?(Circle  the  number  of  your  response.) 

To  a  Great  Extent  Not  at  All 

(5)  (4)  (3)  (2)  (1) 

Comments: 

16.  In  thinking  about  the  MCAS  results  in  your  curriculum  area,  please  share  with  us  what  you  believe  to 
be  the  three  most  important  student  achievement  gaps  or  concerns  identified? 

4. 


Given  these  areas  of  need  for  improvement  in  student  achievement,  please  respond  to  the  following 
questions  about  actions  taken  within  your  curriculum  area  in  the  realms  of  curriculum,  instruction, 
assessment,  and  resource  allocation. 

17.  Have  curriculum  changes  been  made  or  identified  in  your  curriculum  area  to  address  any  of  these 
student  achievement  gaps  or  concerns?     QYES  QNO 
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If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  curriculum  change: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change  (e.g.,  cross-level  team  of  8  English 
Language  Arts  teachers  and  the  curriculum  director)? 


What  results  of  this  change,  if  any,  have  been  observed? 


18.  In  these  or  other  curricular  changes  planned  by  your  district,  please  indicate  any  courses  or  particular 
areas  of  study  within  your  curriculum  focus  that  have  been  eliminated  or  added. 

Courses  eliminated: 

Courses  added: 

Curricular  areas  of  study  eliminated: 

Curricular  areas  of  study  added: 


19.  Have  instructional  changes  been  made  or  identified  in  your  curriculum  area  to  address  any  of  these 
student  achievement  gaps  or  concerns?  QYES  QNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  instructional  change: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 


What  results  of  this  change,  if  any,  have  been  observed? 


20.  Have  changes  in  classroom  assessment  been  made  or  identified  in  your  curriculum  area  to  address 
any  of  these  student  achievement  gaps  or  concerns?  QYES  UlNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  classroom 
assessment: 


What  specific  MCAS  results  prompted  the  change?_ 


Who  was  involved  in  the  decision-making  about  this  change? 
What  results  of  this  change,  if  any,  have  been  observed? 
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21.  Has  the  allocation  of  resources  in  your  curriculum  area  changed  due  to  efforts  to  address  any  of  these 
student  achievement  gaps  or  concerns?  QYES  QNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  allocation  of 
resources: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 


What  results  of  this  change,  if  any,  have  been  observed? 


22.  Please  share  with  us  in  a  few  words  your  own  greatest  challenge  as  a  curriculum  director  in 
promoting  positive,  responsible  use  of  MCAS  data  for  purposes  of  improving  schools  and  increasing 
achievement  for  all  students. 


23.  What  do  you  believe  is  the  most  important  action  you  have  taken  as  a  leader  in  your  district  to 

promote  positive,  responsible  use  of  MCAS  data  for  purposes  of  improving  schools  and  increasing 
achievement  for  all  students? 


BACKGROUND 

Please  note  that  this  information  will  be  used  for  research  purposes  only  and  will  be  kept  strictly 
confidential.  No  individuals  or  districts  will  be  identified  or  recognizable  in  the  final  report. 

Name  of  district: 


Curricular  area  for  which  you  are  responsible: 
Your  title:  


FOLLOW-UP 

□         Please  indicate  here  if  there  is  a  particular  strategy  or  practice  in  terms  of  using  MCAS  data  that 
is  being  used  in  your  district  that  you  think  is  particularly  valuable  and  about  which  you  or 
someone  else  in  your  district  would  be  willing  to  be  interviewed  by  telephone  for  this  study. 

Strategy  or  practice: . 


Name  of  person  to  be  called: 
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Position: 

Telephone  number: 
E-Mail: 


LI         Please  indicate  here  if  you  or  someone  else  in  your  district  would  like  to  provide  additional  input 
to  this  research  study  through  a  telephone  interview 

Name: 


Position: 

Telephone  number: 
E-Mail: 


THANK  YOU  FOR  COMPLETING  THIS  QUESTIONNAIRE! 


Please  return  this  questionnaire  in  the  stamped,  self-addressed  envelope  provided  to: 

Learning  Innovations — A  Division  of  WestEd 

91  Montvale  Avenue 

Stoneham,  MA  02180 
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PRINCIPAL  QUESTIONNAIRE 

Please  note  that  this  study  is  specifically  focused  on  the  use  of  MCAS  results  in  schools  and  districts, 
rather  than  on  related  activities  or  concerns  such  as  general  preparation  for  testing  or  alignment  with 
curriculum  frameworks.  Thank  you  for  taking  the  time  to  complete  this  important  survey. 

1 .    Has  your  school  received  MCAS  results  from  the  district? 
□YES    Qno 

If  YES,  by  when?  (Month: ,  Year: ) 

If  YES,  please  continue  with  question  #2. 

If  NO,  please  explain. __^_ 


If  NO,  please  move  to  Question  #7. 

Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  your  school  from 
your  district  office. 
□District  results 

Please  indicate  the  types  of  breakdowns  provided  in  district  results: 

□By  grade  level 

□By  curriculum  area 

□By  subscores  within  curriculum  areas 

□By  race,  ethnicity 

□By  gender 

□By  school 

□Others  (Please  specify) 

□School  results 

Please  indicate  the  types  of  breakdowns  provided  in  your  individual  school  results: 

□By  grade  level 

□By  curriculum  area 

□By  subscores  within  curriculum  areas 

□By  test  item 

□By  race,  ethnicity 

□By  gender 

□By  feeder  school 

□Others  (Please  specify) 


Please  indicate  the  students  for  which  MCAS  results  were  sent  to  your  school: 
□Results  of  students  who  were  in  the  school  when  tested 
□Results  of  students  who  are  in  your  school  now,  but  were  in  a  feeder  school  when 

tested 
□Other  (Please  specify.) 


Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  individual 
teachers  in  your  school: 

□Results  of  students  they  taught  during  the  year  of  testing 

□Results  of  students  they  currently  teach 

□item  analysis  by  district 

□item  analysis  by  school 

□item  analysis  by  class  group 

□item  analysis  by  student 

□Other  (Please  specify.) 
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3.    Please  indicate  (with  a  check  mark)  which  of  the  following  individuals  or  groups  have  been 
responsible  for  analysis  of  MCAS  data  in  your  school  during  the  current  school  year. 
□Principal 
□Assistant  Principal 
□School  Team  (cross  grade,  cross  subject) 
□Grade  Level  Teams 
□Subject  Area  Teams 
□Other(s)  (Please  specify.) 


51 


4.  Have  you  personally  been  involved  in  analyzing  MCAS  results?  QYES     QNO 

5.  What  computer  software  packages  or  other  data  analysis  tools  (if  any)  are  being  used  in  your  school? 

Software  packages: 

Other  data  analysis  tools: 

6.  What  was  the  primary  method  for  dissemination  of  MCAS  results  to  the  teachers  in  your  school? 


7.    Has  your  superintendent  communicated  with  principals  about  district  expectations  for  use  of  MCAS 
results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource  allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  expectations: 


8.    Does  your  district  provide  any  specific  incentives  for  principals  to  encourage  the  use  of  MCAS 
results  to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  incentives: 


9.    Have  you  communicated  with  teachers  in  your  school  about  district  expectations  for  use  of  MCAS 
results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource  allocation? 
□  YES       □  NO 
If  YES,  please  describe  these  expectations: 


10.  Does  your  district  provide  any  specific  incentives  for  teachers  to  encourage  the  use  of  MCAS  results 
to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 
□  YES       □  NO 
If  YES,  please  describe  these  incentives:  
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11.  What  professional  development  methods  have  been  used  to  provide  teachers  in  your  school  with  the 
knowledge  and  skills  they  need  to  analyze  and  use  MCAS  results  effectively? 

□  District-wide  workshops  for  teachers  (Topics/focus: ) 

□  School-based  workshops  for  teachers  (Topics/focus: ) 

U  Other  (Please  specify.) 

12.  What  methods  have  been  used  to  engage  teachers  in  your  school  in  learning  from  their  own  school's 
MCAS  results  and  using  them  effectively? 

LJ  District-wide  workshops  for  teachers 

LJ  Cross-district  grade  level  working  groups 

LJ  School-based  grade  level  or  subject  area  working  groups 

LJ  Vertical  (across  grades)  working  groups  in  subject  areas 

LJ  School-based,  cross-grade,  and/or  cross-subject  working  groups 

LI  Other  (Please  specify.) 

13.  Please  circle  the  strategy/activity  in  the  checklist  in  #12  above  that  you  have  found  to  be  most 
effective. 

Briefly  explain  why  you  believe  this  strategy/activity  to  be  effective: 


14.  What  (if  any)  outside  resources,  e.g.,  organizations  or  consultants,  has  your  school  or  district  used  to 
help  teachers  to  analyze  and  use  their  MCAS  results  effectively? 


15.  To  what  extent  do  you  believe  that  working  with  MCAS  results  is  serving  as  a  positive  stimulus  for 
teachers  in  your  school  to  work  together  toward  common  purposes  based  on  identified  student 
needs!  (Circle  the  number  of  your  response.) 

To  a  Great  Extent  Not  at  All 

(5)  (4)  (3)  (2)  (1) 

Comments: 

16.  In  thinking  about  the  MCAS  results  in  your  school,  please  share  with  us  what  you  believe  to  be  the 
three  most  important  student  achievement  gaps  or  concerns  identified? 

1.    

2.    

3. 


Given  these  areas  of  need  for  improvement  in  student  achievement,  please  respond  to  the  following 
questions  about  actions  taken  within  your  school  in  the  realms  of  curriculum,  instruction,  assessment,  and 
resource  allocation. 

17.  Have  curriculum  changes  been  made  or  identified  in  your  school  to  address  any  of  these  student 
achievement  gaps  or  concerns?     QYES  QNO 
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If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  curriculum  change: 


What  specific  MCAS  results  prompted  the  change? 

Who  was  involved  in  the  decision-making  about  this  change  (e.g.,  cross-grade  team  of  English 
Language  Arts  teachers  and  the  curriculum  director)? 


What  results  of  this  change,  if  any,  have  been  observed? 


18.  In  these  or  other  curricular  changes  planned  by  your  school,  please  indicate  any  courses  or  particular 
areas  of  study  within  your  school  that  have  been  eliminated  or  added. 

Courses  eliminated: 

Courses  added: 

Curricular  areas  of  study  eliminated: 

Curricular  areas  of  study  added: 


19.  Have  instructional  changes  been  made  or  identified  in  your  school  to  address  any  of  these  student 
achievement  gaps  or  concerns?         QYES  QNO 
If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  instructional  change: 


What  specific  MCAS  results  prompted  the  change?_ 


Who  was  involved  in  the  decision-making  about  this  change*; 


What  results  of  this  change,  if  any,  have  been  observed? 


20.  Have  changes  in  classroom  assessment  been  made  or  identified  in  your  school  to  address  any  of 
these  student  achievement  gaps  or  concerns?^  YES  QNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  classroom 
assessment: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 
What  results  of  this  change,  if  any,  have  been  observed? 
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21.  Has  the  allocation  of  resources  in  your  school  changed  due  to  efforts  to  address  any  of  these  student 
achievement  gaps  or  concerns?         QYES  □NO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  allocation  of 
resources: 


What  specific  MCAS  results  prompted  the  change?_ 


Who  was  involved  in  the  decision-making  about  this  change? 


What  results  of  this  change,  if  any,  have  been  observed? 


22.  Please  share  with  us  in  a  few  words  your  own  greatest  challenge  as  a  school  administrator  in 

promoting  positive,  responsible  use  of  MCAS  data  for  purposes  of  improving  your  school  and 
increasing  achievement  for  all  students. 


23.  What  do  you  believe  is  the  most  important  action  you  have  taken  as  a  leader  in  your  school  to 

promote  positive,  responsible  use  of  MCAS  data  for  purposes  of  improving  your  school  and 
increasing  achievement  for  all  students? 


BACKGROUND 

Please  note  that  this  information  will  be  used  for  research  purposes  only  and  will  be  kept  strictly 
confidential.  No  individuals  or  districts  will  be  identified  or  recognizable  in  the  final  report. 

Name  of  district:  


Name  of  your  school: 


Level  of  your  school:  □Elementary      QMiddle      QHigh 

What  grades  does  your  school  include?  Number  of  students  in  your  school: 

Your  title:  


FOLLOW-UP 

□         Please  indicate  here  if  there  is  a  particular  strategy  or  practice  in  terms  of  using  MCAS  data  that 
is  being  used  in  your  school  or  district  that  you  think  is  particularly  valuable  and  about  which  you 
or  someone  else  in  your  district  would  be  willing  to  be  interviewed  by  telephone  for  this  study. 

Strategy  or  practice: . 


Name  of  person  to  be  called: 
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Position: 

Telephone  number: 
E-Mail: 


Q         Please  indicate  here  if  you  or  someone  else  in  your  school  or  district  would  like  to  provide 
additional  input  to  this  research  study  through  a  telephone  interview 

Name: 

Position: 


Telephone  number: 
E-Mail: 


THANK  YOU  FOR  COMPLETING  THIS  QUESTIONNAIRE! 

Please  return  this  questionnaire  in  the  stamped,  self-addressed  envelope  provided  to: 

Learning  Innovations — A  Division  of  WestEd 

91  Montvale  Avenue 

Stoneham,  MA  02180 
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TEACHER  QUESTIONNAIRE 

Please  note  that  this  study  is  specifically  focused  on  the  use  of  MCAS  results  in  schools  and  districts, 
rather  than  on  related  activities  or  concerns  such  as  general  preparation  for  testing  or  alignment  with 
curriculum  frameworks.  Thank  you  for  taking  the  time  to  complete  this  important  survey. 

1 .    Have  you,  as  a  teacher,  received  MCAS  results  from  your  district  office  or  your  principal? 

□yes   qno 

If  YES,  by  when?  (Month: ,  Year: ) 

If  YES,  please  continue  with  question  #2. 
If  NO,  please  explain. 


If  NO,  please  move  to  Question  #7. 
2.    What  was  the  primary  method  for  dissemination  of  MCAS  results  to  the  teachers  in  your  school? 


3 .    Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  teachers  in  your 
school. 
□District  results 

Please  indicate  the  types  of  breakdowns  provided  in  district  results: 

□By  grade  level 

□By  curriculum  area 

□By  subscores  within  curriculum  areas 

□By  race,  ethnicity 

□By  gender 

□By  school 

□Others  (Please  specify) 

□  School  results 


Please  indicate  the  types  of  breakdowns  provided  in  your  individual  school  results: 
□By  grade  level 
□By  curriculum  area 
□By  subscores  within  curriculum  areas 
□By  test  item 
□By  race,  ethnicity 
□By  gender 
□By  feeder  school 
□Others  (Please  specify) 


Please  indicate  the  students  for  which  MCAS  results  were  given  to  teachers  in  your  school: 
□Results  of  students  who  were  in  the  school  when  tested 
□Results  of  students  who  are  in  your  school  now,  but  were  in  a  feeder  school  when 

tested 
□Other  (Please  specify.) 


Please  indicate  the  level(s)  of  analysis  of  the  MCAS  results  that  were  provided  to  individual 
teachers  in  your  school: 

□Results  of  students  they  taught  during  the  year  of  testing 

□Results  of  students  they  currently  teach 

□item  analysis  by  district 

□item  analysis  by  school 

□item  analysis  by  class  group 

□item  analysis  by  student 
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□Other  (Please  specify.) 


4.  Have  you  personally  been  involved  in  analyzing  MCAS  results?  □  YES     QNO 

5.  Please  indicate  (with  a  check  mark)  which  of  the  following  individuals  or  groups  have  been 
responsible  for  analysis  of  MCAS  data  in  your  school  during  the  current  school  year. 
□Principal 

□Assistant  Principal 

□School  Team  (cross  grade,  cross  subject) 

□Grade  Level  Teams 

□Subject  Area  Teams 

□Other(s)  (Please  specify.) 


6.    What  computer  software  packages  or  other  data  analysis  tools  (if  any)  are  being  used  in  your  school? 
Software  packages: 


Other  data  analysis  tools: 


7.    Has  your  superintendent  communicated  with  teachers  about  district  expectations  for  use  of  MCAS 
results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource  allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  expectations: 


8.  Has  your  principal  communicated  with  teachers  in  your  school  about  expectations  for  use  of  MCAS 
results  to  make  changes  in  curriculum,  instruction,  assessment,  and/or  resource  allocation? 

D  YES       □  NO 

If  YES,  please  describe  these  expectations: 

9.  Does  your  district  provide  any  specific  incentives  for  teachers  to  encourage  the  use  of  MCAS  results 
to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 

□  YES       □  NO 

If  YES,  please  describe  these  incentives: 


10.  Does  your  school  provide  any  specific  incentives  for  teachers  to  encourage  the  use  of  MCAS  results 
to  make  changes  in  instruction,  assessment,  and/or  resource  allocation? 

Q  YES       □  NO 

If  YES,  please  describe  these  incentives: 


1 1 .  What  professional  development  methods  have  been  offered  for  teachers  in  your  school  to  provide 
them  with  the  knowledge  and  skills  they  need  to  analyze  and  use  MCAS  results  effectively? 

Q  District-wide  workshops  for  teachers  (Topics/focus: ) 

O  School-based  workshops  for  teachers  (Topics/focus: ) 

G  Other  (Please  specify.) 
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12.  What  methods  have  been  used  to  engage  teachers  in  your  school  in  learning  from  your  own  school's 
MCAS  results  and  using  them  effectively? 

G  District-wide  workshops  for  teachers 

□  Cross-district  grade  level  working  groups 

LJ  School-based  grade  level  or  subject  area  working  groups 

LJ  Vertical  (across  grades)  working  groups  in  subject  areas 

LJ  School-based,  cross-grade,  and/or  cross-subject  working  groups 

Ql  Other  (Please  specify.) 

13.  Please  circle  the  strategy/activity  in  the  checklist  in  #12  above  that  you  have  found  to  be  most 
effective. 

Briefly  explain  why  you  believe  this  strategy/activity  to  be  effective: 


14.  To  what  extent  do  you  believe  that  working  with  MCAS  results  is  serving  as  a  positive  stimulus  for 
teachers  in  your  school  to  work  together  toward  common  purposes  based  on  identified  student 
needs!  (Circle  the  number  of  your  response.) 

To  a  Great  Extent  Not  at  All 

(5)  (4)  (3)  (2)  (1) 

Comments: 

15.  If  you  have  worked  with  MCAS  results,  to  what  extent  has  this  work  had  an  effect  on  the  choice  of 
instructional  strategies  that  you  use  in  your  own  classroom?  If  you  have  not  engaged  in  work  with 
MCAS  results,  please  write  NONE  here. 

To  a  Great  Extent  Not  at  All 

(5)  (4)  (3)  (2)  (1) 

16.  In  thinking  about  your  own  school's  MCAS  results,  please  share  with  us,  relative  to  your  own 
teaching  responsibilities,  what  you  believe  to  be  the  three  most  important  student  achievement  gaps 
or  concerns  identified?  If  you  have  not  reviewed  results,  please  write  NONE  here. 


8. 


Given  these  areas  of  need  for  improvement  in  student  achievement,  please  respond  to  the  following 
questions  about  actions  you  personally  have  taken  or  actions  that  have  been  taken  by  groups  of  teachers 
within  your  school  or  district  in  the  areas  of  curriculum,  instruction,  and  assessment. 

17.  Have  curriculum  changes  been  made  or  identified  in  your  school  to  address  any  of  these  student 
achievement  gaps  or  concerns?     QYES  QNO 
If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  curriculum  change: 
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What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change  (e.g.,  cross-grade  team  of  English 
Language  Arts  teachers  and  the  curriculum  director)? 


What  results  of  this  change,  if  any,  have  been  observed? 


18.  In  these  or  other  curricular  changes  planned  by  your  school,  please  indicate  any  courses  or  particular 
areas  of  study  within  your  school  that  have  been  eliminated  or  added. 

Courses  eliminated: 

Courses  added: 

Curricular  areas  of  study  eliminated: 

Curricular  areas  of  study  added: 


19.  Have  instructional  changes  been  made  or  identified  in  your  school  to  address  any  of  these  student 
achievement  gaps  or  concerns?         QYES  QNO 
If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  instructional  change: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 


What  results  of  this  change,  if  any,  have  been  observed?_ 


20.  Have  changes  in  classroom  assessment  been  made  or  identified  in  your  school  to  address  any  of 
these  student  achievement  gaps  or  concerns?  □  YES  QNO 

If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  classroom 
assessment: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 
What  results  of  this  change,  if  any,  have  been  observed? 


21.  Has  the  allocation  of  resources  in  your  school  changed  due  to  efforts  to  address  any  of  these  student 
achievement  gaps  or  concerns?         QYES  QNO 

i 
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If  YES,  please  share  ONE  example  of  what  you  believe  to  be  a  significant  change  in  allocation  of 
resources: 


What  specific  MCAS  results  prompted  the  change? 


Who  was  involved  in  the  decision-making  about  this  change? 


What  results  of  this  change,  if  any,  have  been  observed? 


22.  Please  share  with  us  in  a  few  words  your  own  greatest  challenge  as  a  teacher  in  making  positive, 
responsible  use  of  MCAS  data  for  purposes  of  improving  your  school  and  increasing  achievement  for 
your  students. 


23.  What  do  you  believe  is  the  most  important  action  you  have  taken  as  a  leader  in  your  school  to 

promote  positive,  responsible  use  of  MCAS  data  for  purposes  of  improving  your  school  and 
increasing  achievement  for  all  students? 


BACKGROUND 

Please  note  that  this  information  will  be  used  for  research  purposes  only  and  will  be  kept  strictly 
confidential.  No  individuals  or  districts  will  be  identified  or  recognizable  in  the  final  report. 

Name  of  district:  


Name  of  your  school: 


Level  of  your  school:  □Elementary      QMiddle      □High 
Grade(s)  that  you  teach: 


Subject(s)  that  you  teach  (for  middle  school  and  high  school  teachers  or  elementary  specialists): 


FOLLOW-UP 

□         Please  indicate  here  if  there  is  a  particular  strategy  or  practice  in  terms  of  using  MCAS  data  that 
is  being  used  in  your  school  or  district  that  you  think  is  particularly  valuable  and  about  which  you 
or  someone  else  in  your  district  would  be  willing  to  be  interviewed  by  telephone  for  this  study. 

Strategy  or  practice:  __^ 


Name  of  person  to  be  called: 
Position: 


Telephone  number: 
E-Mail:        
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Q         Please  indicate  here  if  you  or  someone  else  in  your  school  or  district  would  like  to  provide 
additional  input  to  this  research  study  through  a  telephone  interview 

Name: 


Position: 

Telephone  number: 
E-Mail: 


THANK  YOU  FOR  COMPLETING  THIS  QUESTIONNAIRE! 

Please  return  this  questionnaire  in  the  stamped,  self-addressed  envelope  provided  to: 

Learning  Innovations — A  Division  of  WestEd 

91  Montvale  Avenue 

Stoneham,  MA  02180 
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DISTRICTS  PARTICIPATING  IN  QUESTIONNAIRE  SURVEY 

ABINGTON 

ACHUSHNET 

ADAMS-CHESHIRE 

AMESBURY 

AMHERST-PELHAM 

ASHBURNHAM-WESTMINSTER  REGIONAL 

ATHOL-ROYALSTON 

AYER 

BARNSTABLE 

BELMONT 

BERKSHIRE  HILLS  REGIONAL 

BERLIN-BOYLSTON 

BEVERLY 

BLACKSTONE-MILLVILE 

BOSTON 

BOURNE 

BOXBOROUGH 

BRAINTREE 

BRIDGEWATER 

BRIMFIELD,  BROCKFIELD,  TOLLAND,  STURBRIDGE,  WALES 

BROCKTON 

BROOKLINE 

BURLINGTON 

CAMBRIDGE 

CANTON 

CARLISLE 

CARVER 

CHELSEA 

CHICOPEE 

CLINTON 

CONCORD/CARLISLE 

DANVERS 

DARTMOUTH 

DENNIS/YARMOUTH 

DIGHTON-REHOBOTH 

DOUGLAS 

DOVER/SHERBORN 

DRACUT 

DUXBURY 

EAST  HAMPTON 

EASTON 

EVERETT 

FALL  RIVER 

FALMOUTH 

FITCHBURG 

FOXBOROUGH 

FRAMINGHAM 

FRANKLIN 
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GARDNER 

GLOUCESTER 

GRAFTON 

GRANBY 

GREENFIELD 

GROTON/DUNSTABLE 

HALIFAX 

HAMILTON-WENHAM 
HAMPDEN-WILBRAHAM 
HAMPSHIRE 

-CHESTERFIELD 

-WESTHAMPTON 

-WILLIAMSBURG 

-SOUTHAMPTON 
HANOVER 
HATFIELD 
HINGHAM 
HOLBROOK 
HOLLISTON 

HOPEDALE  PUBLIC  SCHOOLS 
HOPKINTON 
HUDSON 
HULL 

IPSWICH 

LAWRENCE 

LEICESTER 

LENOX 

LEOMINSTER 

LEXINGTON 

LINCOLN-SUDBURY 

LITTLETON 

LONGMEADOW 

LUDLOW 

LYNN 

LYNN  FIELD 

MANCHESTER 

MARBLEHEAD 

MARION 

MARSHFIELD 

MARTHA'S  VINEYARD 

MASCONOMET 

MATTAPOISETT 

MEDFORD 

MENDON-UPTON 

METHUEN 

MIDDLEBORO 

MILLIS 

MWRSD 
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NATICK 
NAUSET 
NEEDHAM 
NORTH  ADAMS 
NORTHBRIDGE 
NORTHHAMPTON 
NORTH  BERKSHIRE 
NORTH  MIDDLESEX 
NORTH  READING 
NORWELL 
NORWOOD 

OLD  ROCHESTER 

PALMER 
PEABODY 
PITTSFIELD 
PROVINCETOWN 

RANDOLPH 
READING 
REVERE 
ROWE 

SALEM 

SANDWICH 

SAUGUS 

SEEKONK 

SHIRLEY 

SILVER  LAKE 

SOMERVILLE 

SOUTHBRIDGE 

SOUTHERN  BERKSHIRE 

SOUTH  HADLEY 

SOUTHWICK-TOLLAND 

SPRINGFIELD 

STONEHAM 

STOUGHTON 

SWANSEA 

SWAMPSCOTT 


TEWKSBURY 

TOPSFIELD-BOXFORD 

TYNGSBOROUGH 

WACHUSETT 

WAKEFIELD 

WALES 

WALPOLE 

WALTHAM 

WARE 

WATERTOWN 

WEBSTER 

WESTBOROUGH 

WEST  BOYLSTON 

WEST  BRIDGEWATER 
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WESTFIELD 

WESTFORD 

WESTHAMPTON 

WESTON 

WEST  SPRINGFIELD 

WEYMOUTH 

WHITMAN  -  HANSON 

WILLIAMSBURG  PUBLIC  SCHOOLS 

WILMINGTON 

WINCHESTER 

WINCHENDON 

WORCESTER 

WRENTHAM 
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APPENDIX  B  :  List  of  Districts  That  Participated  in 

Focus  Groups 

Andover 

Athol 

Attleboro 

Brookline 

Burlington 

Danvers 

Easthampton 

East  Bridgewater 

Gardner 

Hamilton-Wenham 

Holyoke 

Lawrence 

Longmeadow 

Lowell 

Melrose 

Milbury 

Monson 

Nauset 

North  Andover 

Pentucket 

Plymouth 

Randolph 

Rockland 

Salem 

Scituate 

Southwick-Tolland 

Springfield 

Stoughton 

Sudbury 

Triton 

Wakefield 

Ware 

Westboro 

West  Bridgewater 

Winthrop 

Whitman-Hanson 
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